-
Notifications
You must be signed in to change notification settings - Fork 28.7k
Deflake flutter_desktop_darwin_unittests #104789
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Deduped #106509 into this issue. Notes from @LongCatIsLooong: Running the main runloop in See flutter/engine#33838 (comment). This prevents us from using async expectations in |
Here's an example flake: https://ci.chromium.org/ui/p/flutter/builders/staging/Mac%20Unopt/717/overview It looks like after all tests have run, there's an assert failure:
|
Adds a gtest test fixture mixin and convenience class that instantiates an NSAutoreleasePool at the beginning of each test and flushes it at the end; this allows Objective-C tests using ARC to free allocations at the end of each test. Adds a subclass for the macOS accessibility bridge tests that instantiates and re-uses an NSWindow* across any tests that use it. This is because instantiating, closing, and immediately collecting an NSWindow results in a crash. See: http://www.openradar.me/FB13291861 Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Adds a gtest test fixture mixin and convenience class that instantiates an NSAutoreleasePool at the beginning of each test and flushes it at the end; this allows Objective-C tests using ARC to free allocations at the end of each test. Adds a subclass for the macOS accessibility bridge tests that instantiates and re-uses an NSWindow* across any tests that use it. This is because instantiating, closing, and immediately collecting an NSWindow results in a crash. Prior to this patch, tests started failing (on my machine) around the 855th iteration, and issued the following warning on the 101st iteration: ``` 2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. ( 0 AppKit 0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684 1 AppKit 0x00000001933050e4 _NXCreateWindow + 284 2 AppKit 0x0000000192901ae0 -[NSWindow _commonAwake] + 672 3 AppKit 0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972 4 AppKit 0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796 5 AppKit 0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48 6 flutter_desktop_darwin_unittests 0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328 ``` See: http://www.openradar.me/FB13291861 Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Adds a gtest test fixture mixin and convenience class that instantiates an NSAutoreleasePool at the beginning of each test and flushes it at the end; this allows Objective-C tests using ARC to free allocations at the end of each test. Adds a subclass for the macOS accessibility bridge tests that instantiates and re-uses an NSWindow* across any tests that use it. This is because instantiating, closing, and immediately collecting an NSWindow results in a crash. Prior to this patch, tests started failing (on my machine) around the 855th iteration, and issued the following warning on the 101st iteration: ``` 2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. ( 0 AppKit 0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684 1 AppKit 0x00000001933050e4 _NXCreateWindow + 284 2 AppKit 0x0000000192901ae0 -[NSWindow _commonAwake] + 672 3 AppKit 0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972 4 AppKit 0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796 5 AppKit 0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48 6 flutter_desktop_darwin_unittests 0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328 ``` See: http://www.openradar.me/FB13291861 Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Adds a gtest test fixture mixin and convenience class that instantiates an NSAutoreleasePool at the beginning of each test and flushes it at the end; this allows Objective-C tests using ARC to free allocations at the end of each test. Adds a subclass for the macOS accessibility bridge tests that instantiates and re-uses an NSWindow* across any tests that use it. This is because instantiating, closing, and immediately collecting an NSWindow results in a crash. Prior to this patch, tests started failing (on my machine) around the 855th iteration, and issued the following warning on the 101st iteration: ``` 2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. ( 0 AppKit 0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684 1 AppKit 0x00000001933050e4 _NXCreateWindow + 284 2 AppKit 0x0000000192901ae0 -[NSWindow _commonAwake] + 672 3 AppKit 0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972 4 AppKit 0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796 5 AppKit 0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48 6 flutter_desktop_darwin_unittests 0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328 ``` See: http://www.openradar.me/FB13291861 Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Adds a gtest test fixture mixin and convenience class that instantiates an NSAutoreleasePool at the beginning of each test and flushes it at the end; this allows Objective-C tests using ARC to free allocations at the end of each test. Adds a subclass for the macOS accessibility bridge tests that instantiates and re-uses an NSWindow* across any tests that use it. This is because instantiating, closing, and immediately collecting an NSWindow results in a crash. Prior to this patch, tests started failing (on my machine) around the 855th iteration, and issued the following warning on the 101st iteration: ``` 2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. ( 0 AppKit 0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684 1 AppKit 0x00000001933050e4 _NXCreateWindow + 284 2 AppKit 0x0000000192901ae0 -[NSWindow _commonAwake] + 672 3 AppKit 0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972 4 AppKit 0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796 5 AppKit 0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48 6 flutter_desktop_darwin_unittests 0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328 ``` See: http://www.openradar.me/FB13291861 Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Adds a gtest test fixture mixin and convenience class that instantiates an NSAutoreleasePool at the beginning of each test and flushes it at the end; this allows Objective-C tests using ARC to free allocations at the end of each test. Adds a subclass for the macOS accessibility bridge tests that instantiates and re-uses an NSWindow* across any tests that use it. This is because instantiating, closing, and immediately collecting an NSWindow results in a crash. Prior to this patch, tests started failing (on my machine) around the 855th iteration, and issued the following warning on the 101st iteration: ``` 2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. ( 0 AppKit 0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684 1 AppKit 0x00000001933050e4 _NXCreateWindow + 284 2 AppKit 0x0000000192901ae0 -[NSWindow _commonAwake] + 672 3 AppKit 0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972 4 AppKit 0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796 5 AppKit 0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48 6 flutter_desktop_darwin_unittests 0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328 ``` See: http://www.openradar.me/FB13291861 Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
Wraps all FlutterExternalTexture tests in an autorelease pool to ensure resources are cleaned up between tests. Prior to this change, running these tests via: ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \ --gtest_filter='FlutterEmbedderExternalTextureUnittests.*' Resuling in a segfault: [ERROR:flutter/shell/platform/darwin/graphics/FlutterDarwinContextMetalSkia.mm(35)] Could not create Metal command queue. zsh: segmentation fault ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Wraps all FlutterExternalTexture tests in an autorelease pool to ensure resources are cleaned up between tests. Prior to this change, running these tests via: ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \ --gtest_filter='FlutterEmbedderExternalTextureUnittests.*' Resuling in a segfault: [ERROR:flutter/shell/platform/darwin/graphics/FlutterDarwinContextMetalSkia.mm(35)] Could not create Metal command queue. zsh: segmentation fault ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Wraps all FlutterExternalTexture tests in an autorelease pool to ensure resources are cleaned up between tests. Prior to this change, running these tests via: ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \ --gtest_filter='FlutterEmbedderExternalTextureUnittests.*' Resuling in a segfault: [ERROR:flutter/shell/platform/darwin/graphics/FlutterDarwinContextMetalSkia.mm(35)] Could not create Metal command queue. zsh: segmentation fault ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
Wraps all FlutterTextInputPlugin tests in an autorelease pool to ensure resources are cleaned up between tests. Prior to this change, running these tests via: ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \ --gtest_repeat=1000 \ --gtest_filter='FlutterTextInputPluginTest.TestFirstRectForCharacterRange' Resuling in a test failure: ``` [ RUN ] FlutterTextInputPluginTest.TestFirstRectForCharacterRange 2023-10-31 10:30:48.497112-0700 flutter_desktop_darwin_unittests[5304:4897209] Could not create Metal command queue. 2023-10-31 10:30:48.497761-0700 flutter_desktop_darwin_unittests[5304:4897209] Unable to create FlutterView; no MTLDevice or MTLCommandQueue available. 2023-10-31 10:30:48.520949-0700 flutter_desktop_darwin_unittests[5304:4897209] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Did not record an invocation in OCMStub/OCMExpect/OCMReject. Possible causes are: - The receiver is not a mock object. - The selector conflicts with a selector implemented by OCMStubRecorder/OCMExpectationRecorder.' *** First throw call stack: ( 0 CoreFoundation 0x000000018b1f6800 __exceptionPreprocess + 176 1 libobjc.A.dylib 0x000000018acedeb4 objc_exception_throw + 60 2 CoreFoundation 0x000000018b1f66f0 +[NSException exceptionWithName:reason:userInfo:] + 0 3 flutter_desktop_darwin_unittests 0x00000001039318a0 +[OCMMacroState endStubMacro] + 272 4 flutter_desktop_darwin_unittests 0x0000000100138858 -[FlutterInputPluginTestObjc testFirstRectForCharacterRange] + 1316 5 flutter_desktop_darwin_unittests 0x0000000100152e04 _ZN7flutter7testing62FlutterTextInputPluginTest_TestFirstRectForCharacterRange_Test8TestBodyEv + 60 6 flutter_d *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Did not record an invocation in OCMStub/OCMExpect/OCMReject. Possible causes are: - The receiver is not a mock object. - The selector conflicts with a selector implemented by OCMStubRecorder/OCMExpectationRecorder.' *** First throw call stack: ( 0 CoreFoundation 0x000000018b1f6800 __exceptionPreprocess + 176 1 libobjc.A.dylib 0x000000018acedeb4 objc_exception_throw + 60 2 CoreFoundation 0x000000018b1f66f0 +[NSException exceptionWithName:reason:userInfo:] + 0 3 flutter_desktop_darwin_unittests 0x00000001039318a0 +[OCMMacroState endStubMacro] + 272 4 flutter_desktop_darwin_unittests 0x0000000100138858 -[FlutterInputPluginTestObjc testFirstRectForCharacterRange] + 1316 ``` Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
The tests for FlutterCompositor are not useful. The current tests test two things: 1. That the mocks we set up behave the way we set them up to behave. 2. That the implementation of FlutterCompositor is the current implementation of FlutterCompositor. As an example, consider FlutterCompositorTest.TestPresent: https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L107 Ostensibly, this test verifies that the onPresent callback configured in our fake FlutterViewProvider implementation is called when FlutterCompositor::Present() is called. However, taking a look at the mocking setup: https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L47-L85 We do the following: 1. Mock out FlutterSurfaceManager such that when a surface is requested, we hand back a mock surface. A little gross since we're relying on some knowledge of implementation details of the compositor, but let's take this as reasonable for now. 2. We mock out `FlutterSurface asFlutterMetalTexture` to return a mock texture. Again, we're getting a bit deep into implementation details that the test shouldn't know about, but let's assume this gets us somewhere. 3. We mock out `FlutterSurfaceManager present:notify:` to actually call the `onPresent` callback if it's passed in. In effect, we're testing that: 1. We configured our mock for `FlutterSurfaceManager present:notify:` to call onPresent. 2. That `FlutterCompositor::Present` actually calls `FlutterSurfaceManager present:notify:` despite that being a simple implementation detail of that call. This removes these tests. I have filed the following issue to track refactoring this class for testability and adding tests: flutter/flutter#137648 Encountered these tests as part of deflaking and cleaning up memory allocations throughout the macOS desktop tests. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
The tests for FlutterCompositor are not useful. The current tests test two things: 1. That the mocks we set up behave the way we set them up to behave. 2. That the implementation of FlutterCompositor is the current implementation of FlutterCompositor. As an example, consider FlutterCompositorTest.TestPresent: https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L107 Ostensibly, this test verifies that the onPresent callback configured in our fake FlutterViewProvider implementation is called when FlutterCompositor::Present() is called. However, taking a look at the mocking setup: https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L47-L85 We do the following: 1. Mock out FlutterSurfaceManager such that when a surface is requested, we hand back a mock surface. A little gross since we're relying on some knowledge of implementation details of the compositor, but let's take this as reasonable for now. 2. We mock out `FlutterSurface asFlutterMetalTexture` to return a mock texture. Again, we're getting a bit deep into implementation details that the test shouldn't know about, but let's assume this gets us somewhere. 3. We mock out `FlutterSurfaceManager present:notify:` to actually call the `onPresent` callback if it's passed in. In effect, we're testing that: 1. We configured our mock for `FlutterSurfaceManager present:notify:` to call onPresent. 2. That `FlutterCompositor::Present` actually calls `FlutterSurfaceManager present:notify:` despite that being a simple implementation detail of that call. This removes these tests. I have filed the following issue to track refactoring this class for testability and adding tests: flutter/flutter#137648 Encountered these tests as part of deflaking and cleaning up memory allocations throughout the macOS desktop tests. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
The tests for FlutterCompositor are not useful. The current tests test two things: 1. That the mocks we set up behave the way we set them up to behave. 2. That the implementation of FlutterCompositor is the current implementation of FlutterCompositor. As an example, consider FlutterCompositorTest.TestPresent: https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L107 Ostensibly, this test verifies that the onPresent callback configured in our fake FlutterViewProvider implementation is called when FlutterCompositor::Present() is called. However, taking a look at the mocking setup: https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L47-L85 We do the following: 1. Mock out FlutterSurfaceManager such that when a surface is requested, we hand back a mock surface. A little gross since we're relying on some knowledge of implementation details of the compositor, but let's take this as reasonable for now. 2. We mock out `FlutterSurface asFlutterMetalTexture` to return a mock texture. Again, we're getting a bit deep into implementation details that the test shouldn't know about, but let's assume this gets us somewhere. 3. We mock out `FlutterSurfaceManager present:notify:` to actually call the `onPresent` callback if it's passed in. In effect, we're testing that: 1. We configured our mock for `FlutterSurfaceManager present:notify:` to call onPresent. 2. That `FlutterCompositor::Present` actually calls `FlutterSurfaceManager present:notify:` despite that being a simple implementation detail of that call. This removes these tests. I have filed the following issue to track refactoring this class for testability and adding tests: flutter/flutter#137648 Encountered these tests as part of deflaking and cleaning up memory allocations throughout the macOS desktop tests. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
The tests for FlutterCompositor are not useful. The current tests test two things: 1. That the mocks we set up behave the way we set them up to behave. 2. That the implementation of FlutterCompositor is the current implementation of FlutterCompositor. As an example, consider FlutterCompositorTest.TestPresent: https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L107-L137 Ostensibly, this test verifies that the onPresent callback configured in our fake FlutterViewProvider implementation is called when FlutterCompositor::Present() is called. However, taking a look at the mocking setup: https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L47-L85 We do the following: 1. Mock out FlutterSurfaceManager such that when a surface is requested, we hand back a mock surface. A little gross since we're relying on some knowledge of implementation details of the compositor, but let's take this as reasonable for now. 2. We mock out `FlutterSurface asFlutterMetalTexture` to return a mock texture. Again, we're getting a bit deep into implementation details that the test shouldn't know about, but let's assume this gets us somewhere. 3. We mock out `FlutterSurfaceManager present:notify:` to actually call the `onPresent` callback if it's passed in. In effect, we're testing that: 1. We configured our mock for `FlutterSurfaceManager present:notify:` to call onPresent. 2. That `FlutterCompositor::Present` actually calls `FlutterSurfaceManager present:notify:` despite that being a simple implementation detail of that call. This removes these tests. I have filed the following issue to track refactoring this class for testability and adding tests: flutter/flutter#137648 Encountered these tests as part of deflaking and cleaning up memory allocations throughout the macOS desktop tests. Issue: flutter/flutter#137648 Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
Wraps all FlutterViewController tests in an autorelease pool to ensure resources are cleaned up. Adds a MockFlutterEngineTest subclass of AutoreleasePoolTest that creates an OCPartialMock FlutterEngine and shuts it down at the end of the test. Previously we were not shutting down any FlutterEngine instances we allocated, resulting in potentially thousands of threads and graphics contexts being allocated. Prior to this change, running these tests via: ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \ --gtest_filter='FlutterViewController.*' --gtest_repeat=1000 resulted in test failures and sometimes segfaults. This ensures resources are cleaned up Eventually all unit tests should configure their FlutterEngine via either FlutterEngineTest (which should be an AutoreleasePoolTest) or MockFlutterEngineTest, and the CreateMockFlutterEngine function moved to a static used in the implementation of these. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Wraps all FlutterViewController tests in an autorelease pool to ensure resources are cleaned up. Adds a MockFlutterEngineTest subclass of AutoreleasePoolTest that creates an OCPartialMock FlutterEngine and shuts it down at the end of the test. Previously we were not shutting down any FlutterEngine instances we allocated, resulting in potentially thousands of threads and graphics contexts being allocated. Prior to this change, running these tests via: ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \ --gtest_filter='FlutterViewController.*' --gtest_repeat=1000 resulted in test failures and sometimes segfaults. This ensures resources are cleaned up Eventually all unit tests should configure their FlutterEngine via either FlutterEngineTest (which should be an AutoreleasePoolTest) or MockFlutterEngineTest, and the CreateMockFlutterEngine function moved to a static used in the implementation of these. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Wraps all FlutterViewController tests in an autorelease pool to ensure resources are cleaned up. Adds a MockFlutterEngineTest subclass of AutoreleasePoolTest that creates an OCPartialMock FlutterEngine and shuts it down at the end of the test. Previously we were not shutting down any FlutterEngine instances we allocated, resulting in potentially thousands of threads and graphics contexts being allocated. Prior to this change, running these tests via: ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \ --gtest_filter='FlutterViewController.*' --gtest_repeat=1000 resulted in test failures and sometimes segfaults. This ensures resources are cleaned up Eventually all unit tests should configure their FlutterEngine via either FlutterEngineTest (which should be an AutoreleasePoolTest) or MockFlutterEngineTest, and the CreateMockFlutterEngine function moved to a static used in the implementation of these. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide] and the [C++, Objective-C, Java style guides]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I added new tests to check the change I am making or feature I am adding, or the PR is [test-exempt]. See [testing the engine] for instructions on writing and running engine tests. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I signed the [CLA]. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/wiki/Tree-hygiene#overview [Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene [test-exempt]: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests [Flutter Style Guide]: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style [testing the engine]: https://github.com/flutter/flutter/wiki/Testing-the-engine [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/wiki/Chat
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up. Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up. Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up. Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Also renames the static keyEvent() function CreateKeyEvent, in line with our style guide. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up. Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Also renames the static keyEvent() function CreateKeyEvent, in line with our style guide. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up. Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Also renames the static keyEvent() function CreateKeyEvent, in line with our style guide. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up. Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
Runs all FlutterMenuPlugin tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up. Also extracts out some hackery into the FlutterMenuPluginTest fixture to ensure that NSApplication instantiates everything necessary to do menu bar manipulation. Also replaces the use of OCMock in FlutterMenuPluginTest.mm with a fake FakePluginRegistrar class. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Runs all FlutterMenuPlugin tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up. Also extracts out some hackery into the FlutterMenuPluginTest fixture to ensure that NSApplication instantiates everything necessary to do menu bar manipulation. Also replaces the use of OCMock in FlutterMenuPluginTest.mm with a fake FakePluginRegistrar class. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Also fixes linter warnings about using NSLocalizedString for user-facing strings. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Runs all FlutterMenuPlugin tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up. Also extracts out some hackery into the FlutterMenuPluginTest fixture to ensure that NSApplication instantiates everything necessary to do menu bar manipulation. Also replaces the use of OCMock in FlutterMenuPluginTest.mm with a fake FakePluginRegistrar class. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Also fixes linter warnings about using NSLocalizedString for user-facing strings. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Also replaces the use of OCMock with a fake for tests involving the use of NSPasteboard. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Also replaces the use of OCMock with a fake for tests involving the use of NSPasteboard. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Also replaces the use of OCMock with a fake for tests involving the use of NSPasteboard. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Also replaces the use of OCMock with a fake for tests involving the use of NSPasteboard. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840
Also replaces the use of OCMock with a fake for tests involving the use of NSPasteboard. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable. Issue: flutter/flutter#104789 Issue: flutter/flutter#127441 Issue: flutter/flutter#124840 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
Renamed to capture the broader issue of deflaking the macOS desktop tests, and as part of that work, eliminate the use of OCMock. |
This issue is assigned to @cbracken but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks! |
I'm going to close this since we've made a ton of progress on this and I don't think we're seeing a significant number of flakes on the bots anymore. The underlying issue of dependencies between tests still exists for some tests, but not more than on any other platform. |
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of |
Some tests in
flutter_desktop_darwin_unittests
uses global states, results them not suitable for running parallelly.See: #104748
Using global states for tests are usually not a good idea as it makes tests hard to maintain.
Maybe we can update the how the tests are set up to make them self contained?
The text was updated successfully, but these errors were encountered: