Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Run some unit tests in parallel #33603

Merged
merged 1 commit into from
May 27, 2022
Merged

Run some unit tests in parallel #33603

merged 1 commit into from
May 27, 2022

Conversation

zanderso
Copy link
Member

Uses gtest-parallel to run gtest unit tests in parallel. Adds a dependence on https://chromium.googlesource.com/external/github.com/google/gtest-parallel/, and uses it in run_tests.py. This is a big speedup on my local machine with a lot of cores. It may be less helpful in CI where VMs might have fewer cores, but we'll see.

@zanderso
Copy link
Member Author

On Linux when I run this with:

$ ./flutter/testing/run_tests.py --variant host_debug_unopt --type engine --engine-capture-core-dump --use-sanitizer-suppressions

I consistently get the following failure:

[ RUN      ] ShellTest.OnPlatformViewDestroyWhenThreadsAreMerging
[FATAL:flutter/shell/common/rasterizer.cc(526)] Check failed: !external_view_embedder_->GetUsedThisFrame().
[ERROR:flutter/fml/backtrace.cc(116)] Caught signal SIGABRT during program execution.
Frame 0: 0x7f7f66e73546 abort
Frame 1: 0x55d441890dd9 Unknown
Frame 2: 0x55d441890db8 fml::LogMessage::~LogMessage()
Frame 3: 0x55d4417778f5 flutter::Rasterizer::DrawToSurfaceUnsafe()
Frame 4: 0x55d441775891 flutter::Rasterizer::DrawToSurface()
Frame 5: 0x55d441776db0 flutter::Rasterizer::DoDraw()
Frame 6: 0x55d44177af72 flutter::Rasterizer::Draw()::$_2::operator()()
Frame 7: 0x55d44177aec2 std::__1::__invoke<>()
Frame 8: 0x55d44177ae5d std::__1::__invoke_void_return_wrapper<>::__call<>()
Frame 9: 0x55d44177ae35 std::__1::__function::__alloc_func<>::operator()()
Frame 10: 0x55d44177a461 std::__1::__function::__func<>::operator()()
Frame 11: 0x55d4414ff39a std::__1::__function::__value_func<>::operator()()
Frame 12: 0x55d4414ff335 std::__1::function<>::operator()()
Frame 13: 0x55d4417836e5 flutter::Pipeline<>::Consume()
Frame 14: 0x55d441775cbc flutter::Rasterizer::Draw()
Frame 15: 0x55d4417c7e44 flutter::Shell::OnAnimatorDraw()::$_12::operator()()
Frame 16: 0x55d4417c7c81 fml::internal::CopyableLambda<>::operator()<>()
Frame 17: 0x55d4417c7c55 std::__1::__invoke<>()
Frame 18: 0x55d4417c7c15 std::__1::__invoke_void_return_wrapper<>::__call<>()
Frame 19: 0x55d4417c7bed std::__1::__function::__alloc_func<>::operator()()
Frame 20: 0x55d4417c7109 std::__1::__function::__func<>::operator()()
Frame 21: 0x55d4414e9222 std::__1::__function::__value_func<>::operator()()
Frame 22: 0x55d4414e91e5 std::__1::function<>::operator()()
Frame 23: 0x55d44189378e fml::MessageLoopImpl::FlushTasks()
Frame 24: 0x55d44189368a fml::MessageLoopImpl::RunExpiredTasksNow()
Frame 25: 0x55d4418aa06a fml::MessageLoopLinux::OnEventFired()
Frame 26: 0x55d4418aa01c fml::MessageLoopLinux::Run()
Frame 27: 0x55d441893629 fml::MessageLoopImpl::DoRun()
Frame 28: 0x55d4418927bd fml::MessageLoop::Run()
Frame 29: 0x55d4418a642e fml::Thread::Thread()::$_0::operator()()
Frame 30: 0x55d4418a6395 std::__1::__invoke<>()
Frame 31: 0x55d4418a633d std::__1::__thread_execute<>()
Frame 32: 0x55d4418a6128 std::__1::__thread_proxy<>()
Frame 33: 0x7f7f67177d80 start_thread
Frame 34: 0x7f7f66f4b76f __clone

cc @cyanglaz

@cyanglaz
Copy link
Contributor

cyanglaz commented May 25, 2022 •
edited
Loading

I didn't see the failure mentioned in #33603 (comment)

But I saw:

[ RUN      ] FlutterMenuPluginTest.TestSetMenu
2022-05-25 09:57:57.296 flutter_desktop_darwin_unittests[60836:1899166] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[__NSCFConstantString stringByReplacingOccurrencesOfString:withString:options:range:]: nil argument'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007ff801bcf7b3 __exceptionPreprocess + 242
	1   libobjc.A.dylib                     0x00007ff80192fbc3 objc_exception_throw + 48
	2   Foundation                          0x00007ff80298440f -[NSString stringByReplacingOccurrencesOfString:withString:options:range:] + 276
	3   Foundation                          0x00007ff8029842d0 -[NSString stringByReplacingOccurrencesOfString:withString:] + 97
	4   flutter_desktop_darwin_unittests    0x00000001047260ac -[FlutterMenuPlugin menuItemFromFlutterRepresentation:] + 1884
	5   flutter_desktop_darwin_unittests    0x00000001047298f0 -[FlutterMenuPlugin setMenus:] + 512
	6   flutter_desktop_darwin_unittests    0x0000000104729601 -[FlutterMenuPlugin handleMethodCall:result:] + 561
	7   flutter_desktop_darwin_unittests    0x000000010467d3ff -[FlutterMenuPluginTestObjc testSetMenu] + 8863
	8   flutter_desktop_darwin_unittests    0x0000000104682303 _ZN7flutter7testing38FlutterMenuPluginTest_TestSetMenu_Test8TestBodyEv + 67
	9   flutter_desktop_darwin_unittests    0x0000000107719609 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc + 137
	10  flutter_desktop_darwin_unittests    0x00000001076eeb25 _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc + 117
	11  flutter_desktop_darwin_unittests    0x00000001076eea41 _ZN7testing4Test3RunEv + 209
	12  flutter_desktop_darwin_unittests    0x00000001076ef7d2 _ZN7testing8TestInfo3RunEv + 290
	13  flutter_desktop_darwin_unittests    0x00000001076f0c5b _ZN7testing9TestSuite3RunEv + 331
	14  flutter_desktop_darwin_unittests    0x00000001076fdb3d _ZN7testing8internal12UnitTestImpl11RunAllTestsEv + 973
	15  flutter_desktop_darwin_unittests    0x000000010771fbe9 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc + 137
	16  flutter_desktop_darwin_unittests    0x00000001076fd6d5 _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc + 117
	17  flutter_desktop_darwin_unittests    0x00000001076fd56a _ZN7testing8UnitTest3RunEv + 186
	18  flutter_desktop_darwin_unittests    0x00000001048af7d3 _Z13RUN_ALL_TESTSv + 35
	19  flutter_desktop_darwin_unittests    0x00000001048af6e3 main + 547
	20  dyld                                0x000000011d1c751e start + 462
)
libc++abi: terminating with uncaught exception of type NSException
[ERROR:flutter/fml/backtrace.cc(116)] Caught signal SIGABRT during program execution.
Frame 0: 0x7ff8019d8d24 abort
Frame 1: 0x7ff801a49082 abort_message
Frame 2: 0x7ff801a3a1bd demangling_unexpected_handler()
Frame 3: 0x7ff801936e11 _objc_terminate()
Frame 4: 0x7ff801a484a7 std::__terminate()
Frame 5: 0x7ff801a4ad05 __cxa_get_exception_ptr
Frame 6: 0x7ff801a4accc __cxxabiv1::exception_cleanup_func()
Frame 7: 0x7ff80192fcc1 objc_exception_throw
Frame 8: 0x7ff80298440f -[NSString stringByReplacingOccurrencesOfString:withString:options:range:]
Frame 9: 0x7ff8029842d0 -[NSString stringByReplacingOccurrencesOfString:withString:]
Frame 10: 0x1047260ac -[FlutterMenuPlugin menuItemFromFlutterRepresentation:]
Frame 11: 0x1047298f0 -[FlutterMenuPlugin setMenus:]
Frame 12: 0x104729601 -[FlutterMenuPlugin handleMethodCall:result:]
Frame 13: 0x10467d3ff -[FlutterMenuPluginTestObjc testSetMenu]
Frame 14: 0x104682303 flutter::testing::FlutterMenuPluginTest_TestSetMenu_Test::TestBody()
Frame 15: 0x107719609 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
Frame 16: 0x1076eeb25 testing::internal::HandleExceptionsInMethodIfSupported<>()
Frame 17: 0x1076eea41 testing::Test::Run()
Frame 18: 0x1076ef7d2 testing::TestInfo::Run()
Frame 19: 0x1076f0c5b testing::TestSuite::Run()
Frame 20: 0x1076fdb3d testing::internal::UnitTestImpl::RunAllTests()
Frame 21: 0x10771fbe9 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
Frame 22: 0x1076fd6d5 testing::internal::HandleExceptionsInMethodIfSupported<>()
Frame 23: 0x1076fd56a testing::UnitTest::Run()
Frame 24: 0x1048af7d3 RUN_ALL_TESTS()
Frame 25: 0x1048af6e3 main
Frame 26: 0x11d1c751e start

Maybe another flaky tests that is surfaced by running parallel?

@zanderso
Copy link
Member Author

@cyanglaz yeah, that's probably a different flaky test. To see the flaky shell test, maybe just comment out flutter_desktop_darwin_unittests?

@cyanglaz
Copy link
Contributor

cyanglaz commented May 25, 2022 •
edited
Loading

Weird. I commented out everything except ('shell_unittests', repeat_flags), and the tests that ran for me were not the same as in https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8813227694034149665/+/u/test:_Host_Tests_for_host_debug_unopt/stdout?format=raw
And none of the ShellTests.xxx ran for me.

For example i had something like:

[ RUN      ] Play/EntityTest.SolidStrokeCoverageIsCorrect/OpenGLES
[       OK ] Play/EntityTest.SolidStrokeCoverageIsCorrect/OpenGLES (8 ms)

But they weren't in the tests ran on CI.

@zanderso
Copy link
Member Author

@cyanglaz It sounds like you might have commented out too much. Maybe you commented out https://github.com/flutter/engine/pull/33603/files#diff-521d7c59c3f721b259f094c760c2613e16e889b0f24702280924466109e3b0b0R235 ?

@cyanglaz
Copy link
Contributor

Oh there were additional tests ran for mac, I commented those out now.
I saw all the tests passing, the output:

log

ychris-macpro2:src ychris$ ./flutter/testing/run_tests.py --variant host_debug_unopt --type engine --engine-capture-core-dump --use-sanitizer-suppressions
Running Engine Unit-tests.
Running shell_unittests in /Users/ychris/flutter/engine/src

Running command "python3 /Users/ychris/flutter/engine/src/third_party/gtest-parallel/gtest-parallel /Users/ychris/flutter/engine/src/out/host_debug_unopt/shell_unittests --repeat=2"
[260/260] ShellTest.InitializeWithDisabledGpu (661 ms)

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Command run successfully in 2.67 seconds: python3 /Users/ychris/flutter/engine/src/third_party/gtest-parallel/gtest-parallel /Users/ychris/flutter/engine/src/out/host_debug_unopt/shell_unittests --repeat=2

Running command "python3 test.py"
Using font subset binary at /Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset (/Users/ychris/flutter/engine/src/out/host_debug_unopt/zip_archives/darwin-x64/font-subset.zip)
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset /Users/ychris/flutter/engine/src/flutter/tools/font-subset/gen/1.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "57347"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset /Users/ychris/flutter/engine/src/flutter/tools/font-subset/gen/1.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "0xE003"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset /Users/ychris/flutter/engine/src/flutter/tools/font-subset/gen/1.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "\uE003"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset /Users/ychris/flutter/engine/src/flutter/tools/font-subset/gen/1.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "57348"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset /Users/ychris/flutter/engine/src/flutter/tools/font-subset/gen/2.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "0xE003 0xE004"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset /Users/ychris/flutter/engine/src/flutter/tools/font-subset/gen/2.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "0xE003 0xE004 57347"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset /Users/ychris/flutter/engine/src/flutter/tools/font-subset/gen/3.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "0xE003 0xE004 0xE021"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset output.ttf does-not-exist.ttf
STDIN: "1"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset output.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "0xFFFFFFFF"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset output.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "-1"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset output.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "foo"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset output.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "0xE003 0x12 0xE004"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset non-existent-dir/output.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: "0xE003"
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset output.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: " "
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset output.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: ""
Success.
Running command:
/Users/ychris/flutter/engine/src/out/host_debug_unopt/font-subset output.ttf /Users/ychris/flutter/engine/src/flutter/tools/font-subset/fixtures/MaterialIcons-Regular.ttf
STDIN: ""
Success.
All tests passed

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Command run successfully in 0.25 seconds: python3 test.py

@zanderso
Copy link
Member Author

Ah okay. I mean the failure seems to be pretty timing dependent, so getting it to trigger could very well require all the rest of the tests to be enabled as well (except for the other flaking ones).

@cyanglaz cyanglaz mentioned this pull request May 25, 2022
Merged
8 tasks
@zanderso zanderso force-pushed the parallelize-some-tests branch from 4b71425 to 0797a27 Compare May 25, 2022 23:30
@zanderso zanderso mentioned this pull request May 26, 2022
Merged
@zanderso zanderso force-pushed the parallelize-some-tests branch from 0797a27 to cf78340 Compare May 26, 2022 15:40
@cyanglaz cyanglaz mentioned this pull request May 26, 2022
Closed
@chinmaygarde
Copy link
Member

We can't land this just yet as there are multiple flakes revealed by this patch. Moving to WIP for now.

@chinmaygarde chinmaygarde added the Work in progress (WIP) Not ready (yet) for review! label May 26, 2022
@cyanglaz cyanglaz mentioned this pull request May 26, 2022
Closed
@zanderso zanderso force-pushed the parallelize-some-tests branch 2 times, most recently from dec63b2 to 4521833 Compare May 27, 2022 01:47
cyanglaz
cyanglaz reviewed May 27, 2022
View reviewed changes
testing/run_tests.py
if IsMac():
# flutter_desktop_darwin_unittests uses global state that isn't handled
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created an issue: flutter/flutter#104789 to track a potential work to make the tests avoid using global states. Maybe we can add a reference to the issue here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@zanderso zanderso force-pushed the parallelize-some-tests branch from 4521833 to c51acc0 Compare May 27, 2022 02:04
@zanderso zanderso merged commit 50b3a6e into flutter:main May 27, 2022
@zanderso zanderso deleted the parallelize-some-tests branch May 27, 2022 03:14
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request May 27, 2022
Merged
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 27, 2022
@engine-flutter-autoroll
50b3a6e Run some unit tests in parallel (flutter/engine#33603)
74f9166
@zanderso
Copy link
Member Author

Linux Unopt is down to under 30 minutes from over 50 minutes after this =)

houhuayong pushed a commit to houhuayong/engine that referenced this pull request Jun 21, 2022
@zanderso @houhuayong
Run some unit tests in parallel (flutter#33603)
9312f82
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@cyanglaz cyanglaz cyanglaz left review comments

@chinmaygarde chinmaygarde chinmaygarde approved these changes

Assignees
No one assigned
Labels
Work in progress (WIP) Not ready (yet) for review!
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zanderso @cyanglaz @chinmaygarde