Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix: structured inputs for CudaGraphsTorchTensorRTModule#3407

Merged
zewenli98 merged 3 commits into
pytorch:mainfrom
keehyuna:kwargs_cudagraphs
Mar 21, 2025
Merged

fix: structured inputs for CudaGraphsTorchTensorRTModule#3407
zewenli98 merged 3 commits into
pytorch:mainfrom
keehyuna:kwargs_cudagraphs

Conversation

@keehyuna
Copy link
Copy Markdown
Collaborator

@keehyuna keehyuna commented Feb 24, 2025

Description

Add support for structured inputs in CudaGraphsTorchTensorRTModule using torch's tree utilities to properly handle both args and kwargs input

Fixes #3406

Type of change

Please delete options that are not relevant and/or add your own.

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

@github-actions github-actions Bot added component: api [Python] Issues re: Python API component: runtime component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Feb 24, 2025
@github-actions github-actions Bot requested a review from gs-olive February 24, 2025 14:06
@keehyuna keehyuna self-assigned this Feb 24, 2025
@github-actions github-actions Bot added the component: tests Issues re: Tests label Feb 25, 2025
@keehyuna keehyuna requested review from cehongwang, narendasan and peri044 and removed request for gs-olive February 25, 2025 05:43
@keehyuna keehyuna marked this pull request as ready for review February 25, 2025 05:43
narendasan
narendasan reviewed Feb 28, 2025
View reviewed changes
Comment thread tests/py/dynamo/runtime/test_004_weight_streaming.py Outdated
keehyuna
keehyuna commented Mar 2, 2025
View reviewed changes
Comment thread tests/py/dynamo/runtime/test_004_weight_streaming.py
("cpp_runtime", False),
]
)
@unittest.skipIf(
Copy link
Copy Markdown
Collaborator Author

@keehyuna keehyuna Mar 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test took 80sec on rtx4080

Copy link
Copy Markdown
Collaborator

@narendasan narendasan Mar 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to create a new runner instance for tests like these if you feel its taking too long

Copy link
Copy Markdown
Collaborator Author

@keehyuna keehyuna Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it looks fine. according to log, it was running under Test torch compile backend test, which took 12 minutes.

peri044
peri044 approved these changes Mar 7, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@peri044 peri044 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@narendasan
Copy link
Copy Markdown
Collaborator

narendasan commented Mar 10, 2025

@keehyuna can you resolve the conflict then we can merge?

@keehyuna keehyuna force-pushed the kwargs_cudagraphs branch from 16d3ac8 to 9ec010a Compare March 11, 2025 01:06
@keehyuna keehyuna force-pushed the kwargs_cudagraphs branch from 9ec010a to 77c4241 Compare March 18, 2025 08:26
zewenli98
zewenli98 approved these changes Mar 21, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@zewenli98 zewenli98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@zewenli98 zewenli98 merged commit 35c489d into pytorch:main Mar 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@narendasan narendasan narendasan left review comments

@peri044 peri044 peri044 approved these changes

@zewenli98 zewenli98 zewenli98 approved these changes

@cehongwang cehongwang Awaiting requested review from cehongwang

Assignees

@keehyuna keehyuna

Labels

cla signed component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: runtime component: tests Issues re: Tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

🐛 [Bug] Encountered bug when using Torch-TensorRT

5 participants

@keehyuna @narendasan @peri044 @zewenli98 @facebook-github-bot