Thanks to visit codestin.com
Credit goes to github.com

Skip to content

DISABLED AotInductorTest.BasicTestCuda (build.bin.test_aoti_inference) #152888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pytorch-bot bot opened this issue May 6, 2025 · 3 comments
Open

DISABLED AotInductorTest.BasicTestCuda (build.bin.test_aoti_inference) #152888

pytorch-bot bot opened this issue May 6, 2025 · 3 comments
Labels
high priority module: aotinductor aot inductor module: flaky-tests Problem is a flaky test in CI oncall: pt2 skipped Denotes a (flaky) test currently skipped in CI. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@pytorch-bot
Copy link

pytorch-bot bot commented May 6, 2025 •
edited
Loading

Platforms: inductor

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

Over the past 3 hours, it has been determined flaky in 11 workflow(s) with 22 failures and 11 successes.

Debugging instructions (after clicking on the recent samples link):
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:

  1. Click on the workflow logs linked above
  2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
  3. Grep for AotInductorTest.BasicTestCuda
  4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.
Sample error message 
unknown file
C++ exception with description "Error in dlopen: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /tmp/torchinductor_jenkins/cycbfqzvaesb562wvzdixz3vb5gax4oe6ys7kwovlnnc2ceecyrz/cg3lzvlvcuugtygg5rm7epgdxgoeritrbg7b5ufyto5npb4wt3pt.wrapper.so)
Exception raised from DynamicLibrary at ../aten/src/ATen/DynamicLibrary.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9c (0x7f142f6e111c in /var/lib/jenkins/workspace/build/lib/libc10.so)
frame #1: <unknown function> + 0xdda719 (0x7f1446bda719 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #2: torch::inductor::AOTIModelContainerRunner::AOTIModelContainerRunner(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) + 0x123 (0x7f144bc1b653 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #3: torch::inductor::AOTIModelContainerRunnerCuda::AOTIModelContainerRunnerCuda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) + 0x11 (0x7f1433929811 in /var/lib/jenkins/workspace/build/lib/libtorch_cuda.so)
frame #4: std::_MakeUniq<torch::inductor::AOTIModelContainerRunnerCuda>::__single_object std::make_unique<torch::inductor::AOTIModelContainerRunnerCuda, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x95 (0x555ab9e244f5 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #5: <unknown function> + 0x333eb (0x555ab9e133eb in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #6: torch::aot_inductor::AotInductorTest_BasicTestCuda_Test::TestBody() + 0x46 (0x555ab9e13796 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #7: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) + 0x51 (0x555ab9e66a81 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #8: <unknown function> + 0x768b0 (0x555ab9e568b0 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #9: testing::TestInfo::Run() + 0x40a (0x555ab9e56dca in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #10: <unknown function> + 0x7aea9 (0x555ab9e5aea9 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #11: testing::internal::UnitTestImpl::RunAllTests() + 0xf28 (0x555ab9e5c2f8 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #12: testing::UnitTest::Run() + 0x93 (0x555ab9e5cac3 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #13: main + 0x104 (0x555ab9e0fc04 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #14: __libc_start_main + 0xf3 (0x7f142ee8a083 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #15: _start + 0x2e (0x555ab9e118de in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
" thrown in the test body.
unknown file:0: C++ failure

Test file path: `` or test/run_test

ResponseTimeoutError: Response timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/main/test/ -1 (connected: true, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0)
headers: {}

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @clee2000 @chauhang @penguinwu @desertfire @chenyang78 @yushangdi @benjaminglass1 @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan @angelayi @suo @ydwu4
@pytorch-bot pytorch-bot bot added module: flaky-tests Problem is a flaky test in CI skipped Denotes a (flaky) test currently skipped in CI. oncall: pt2 labels May 6, 2025
@pytorch-bot PyTorch Bot
Copy link
Author

pytorch-bot bot commented May 6, 2025

Hello there! From the DISABLED prefix in this issue title, it looks like you are attempting to disabled a job in PyTorch CI. The information I have parsed is below:
  • Job name: AotInductorTest.BasicTestCuda (build.bin.test_aoti_inference)
  • Credential: pytorch-bot[bot]

Within ~15 minutes, AotInductorTest.BasicTestCuda (build.bin.test_aoti_inference) and all of its dependants will be disabled in PyTorch CI. Please verify that the job name looks correct. With great power comes great responsibility.

@pytorch-bot PyTorch Bot
Copy link
Author

pytorch-bot bot commented May 6, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@masnesral masnesral added the module: aotinductor aot inductor label May 6, 2025
@masnesral masnesral added high priority triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module and removed oncall: export labels May 6, 2025
@masnesral
Copy link
Contributor

hi-pri per oncall runbook

@yushangdi yushangdi mentioned this issue May 7, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
high priority module: aotinductor aot inductor module: flaky-tests Problem is a flaky test in CI oncall: pt2 skipped Denotes a (flaky) test currently skipped in CI. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@masnesral