Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Deflake flutter_desktop_darwin_unittests #104789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cyanglaz opened this issue May 27, 2022 · 6 comments
Closed

Deflake flutter_desktop_darwin_unittests #104789

cyanglaz opened this issue May 27, 2022 · 6 comments
Labels
engine flutter/engine repository. See also e: labels. P2 Important issues not at the top of the work list platform-mac Building on or for macOS specifically team-engine Owned by Engine team triaged-engine Triaged by Engine team

Comments

@cyanglaz
Copy link
Contributor

Some tests in flutter_desktop_darwin_unittests uses global states, results them not suitable for running parallelly.

See: #104748

Using global states for tests are usually not a good idea as it makes tests hard to maintain.
Maybe we can update the how the tests are set up to make them self contained?

@cyanglaz cyanglaz added platform-mac Building on or for macOS specifically P2 Important issues not at the top of the work list engine flutter/engine repository. See also e: labels. labels May 27, 2022
@cbracken
Copy link
Member

Deduped #106509 into this issue. Notes from @LongCatIsLooong:

Running the main runloop in kCFRunLoopDefaultMode in a test causes the test to crash.

See flutter/engine#33838 (comment).

This prevents us from using async expectations in flutter_desktop_darwin_unittests.

@zanderso
Copy link
Member

Here's an example flake: https://ci.chromium.org/ui/p/flutter/builders/staging/Mac%20Unopt/717/overview

It looks like after all tests have run, there's an assert failure:

[FATAL:flutter/shell/common/shell.cc(1557)] Check failed: task_runners_.GetUITaskRunner()->RunsTasksOnCurrentThread().

@flutter-triage-bot flutter-triage-bot bot added team-engine Owned by Engine team triaged-engine Triaged by Engine team labels Jul 8, 2023
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 26, 2023
Adds a gtest test fixture mixin and convenience class that instantiates
an NSAutoreleasePool at the beginning of each test and flushes it at the
end; this allows Objective-C tests using ARC to free  allocations at the
end of each test.

Adds a subclass for the macOS accessibility bridge tests that
instantiates and re-uses an NSWindow* across any tests that use it. This
is because instantiating, closing, and immediately collecting an
NSWindow results in a crash.

See: http://www.openradar.me/FB13291861

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 26, 2023
Adds a gtest test fixture mixin and convenience class that instantiates
an NSAutoreleasePool at the beginning of each test and flushes it at the
end; this allows Objective-C tests using ARC to free  allocations at the
end of each test.

Adds a subclass for the macOS accessibility bridge tests that
instantiates and re-uses an NSWindow* across any tests that use it. This
is because instantiating, closing, and immediately collecting an
NSWindow results in a crash.

Prior to this patch, tests started failing (on my machine) around the
855th iteration, and issued the following warning on the 101st
iteration:

```
2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. (
  0   AppKit                              0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684
  1   AppKit                              0x00000001933050e4 _NXCreateWindow + 284
  2   AppKit                              0x0000000192901ae0 -[NSWindow _commonAwake] + 672
  3   AppKit                              0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972
  4   AppKit                              0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796
  5   AppKit                              0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48
  6   flutter_desktop_darwin_unittests    0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328
```

See: http://www.openradar.me/FB13291861

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 26, 2023
Adds a gtest test fixture mixin and convenience class that instantiates
an NSAutoreleasePool at the beginning of each test and flushes it at the
end; this allows Objective-C tests using ARC to free  allocations at the
end of each test.

Adds a subclass for the macOS accessibility bridge tests that
instantiates and re-uses an NSWindow* across any tests that use it. This
is because instantiating, closing, and immediately collecting an
NSWindow results in a crash.

Prior to this patch, tests started failing (on my machine) around the
855th iteration, and issued the following warning on the 101st
iteration:

```
2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. (
  0   AppKit                              0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684
  1   AppKit                              0x00000001933050e4 _NXCreateWindow + 284
  2   AppKit                              0x0000000192901ae0 -[NSWindow _commonAwake] + 672
  3   AppKit                              0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972
  4   AppKit                              0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796
  5   AppKit                              0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48
  6   flutter_desktop_darwin_unittests    0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328
```

See: http://www.openradar.me/FB13291861

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 26, 2023
Adds a gtest test fixture mixin and convenience class that instantiates
an NSAutoreleasePool at the beginning of each test and flushes it at the
end; this allows Objective-C tests using ARC to free  allocations at the
end of each test.

Adds a subclass for the macOS accessibility bridge tests that
instantiates and re-uses an NSWindow* across any tests that use it. This
is because instantiating, closing, and immediately collecting an
NSWindow results in a crash.

Prior to this patch, tests started failing (on my machine) around the
855th iteration, and issued the following warning on the 101st
iteration:

```
2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. (
  0   AppKit                              0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684
  1   AppKit                              0x00000001933050e4 _NXCreateWindow + 284
  2   AppKit                              0x0000000192901ae0 -[NSWindow _commonAwake] + 672
  3   AppKit                              0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972
  4   AppKit                              0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796
  5   AppKit                              0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48
  6   flutter_desktop_darwin_unittests    0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328
```

See: http://www.openradar.me/FB13291861

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 26, 2023
Adds a gtest test fixture mixin and convenience class that instantiates
an NSAutoreleasePool at the beginning of each test and flushes it at the
end; this allows Objective-C tests using ARC to free  allocations at the
end of each test.

Adds a subclass for the macOS accessibility bridge tests that
instantiates and re-uses an NSWindow* across any tests that use it. This
is because instantiating, closing, and immediately collecting an
NSWindow results in a crash.

Prior to this patch, tests started failing (on my machine) around the
855th iteration, and issued the following warning on the 101st
iteration:

```
2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. (
  0   AppKit                              0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684
  1   AppKit                              0x00000001933050e4 _NXCreateWindow + 284
  2   AppKit                              0x0000000192901ae0 -[NSWindow _commonAwake] + 672
  3   AppKit                              0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972
  4   AppKit                              0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796
  5   AppKit                              0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48
  6   flutter_desktop_darwin_unittests    0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328
```

See: http://www.openradar.me/FB13291861

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
auto-submit bot pushed a commit to flutter/engine that referenced this issue Oct 26, 2023
Adds a gtest test fixture mixin and convenience class that instantiates an NSAutoreleasePool at the beginning of each test and flushes it at the end; this allows Objective-C tests using ARC to free  allocations at the end of each test.

Adds a subclass for the macOS accessibility bridge tests that instantiates and re-uses an NSWindow* across any tests that use it. This is because instantiating, closing, and immediately collecting an NSWindow results in a crash.

Prior to this patch, tests started failing (on my machine) around the 855th iteration, and issued the following warning on the 101st iteration:

```
2023-10-26 13:02:45.390829-0700 flutter_desktop_darwin_unittests[40837:1509026] [Window] WARNING: NSWindow has detected an excessive live window count of 101. Window 0x1423 of class 'NSWindow' created after passing the threshold of 100. This window is not necessarily the cause, and this warning will only be shown once per window class. (
  0   AppKit                              0x0000000192820d28 -[NSWindow _setWindowNumber:] + 684
  1   AppKit                              0x00000001933050e4 _NXCreateWindow + 284
  2   AppKit                              0x0000000192901ae0 -[NSWindow _commonAwake] + 672
  3   AppKit                              0x000000019281ff00 -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 972
  4   AppKit                              0x000000019281f798 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 796
  5   AppKit                              0x000000019281f470 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 48
  6   flutter_desktop_darwin_unittests    0x0000000100001e3c _ZN7flutter7testing89AccessibilityBridgeMacTest_SendsAccessibilityCreateNotificationToWindowOfFlutterView_Test8TestBodyEv + 328
```

See: http://www.openradar.me/FB13291861

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840

[C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 30, 2023
Wraps all FlutterExternalTexture tests in an autorelease pool to ensure
resources are cleaned up between tests.

Prior to this change, running these tests via:

    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \
        --gtest_filter='FlutterEmbedderExternalTextureUnittests.*'

Resuling in a segfault:

    [ERROR:flutter/shell/platform/darwin/graphics/FlutterDarwinContextMetalSkia.mm(35)]
    Could not create Metal command queue.
    zsh: segmentation fault
    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 30, 2023
Wraps all FlutterExternalTexture tests in an autorelease pool to ensure
resources are cleaned up between tests.

Prior to this change, running these tests via:

    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \
        --gtest_filter='FlutterEmbedderExternalTextureUnittests.*'

Resuling in a segfault:

    [ERROR:flutter/shell/platform/darwin/graphics/FlutterDarwinContextMetalSkia.mm(35)]
    Could not create Metal command queue.
    zsh: segmentation fault
    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
auto-submit bot pushed a commit to flutter/engine that referenced this issue Oct 30, 2023
Wraps all FlutterExternalTexture tests in an autorelease pool to ensure resources are cleaned up between tests.

Prior to this change, running these tests via:

    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \
        --gtest_filter='FlutterEmbedderExternalTextureUnittests.*'

Resuling in a segfault:

    [ERROR:flutter/shell/platform/darwin/graphics/FlutterDarwinContextMetalSkia.mm(35)]
    Could not create Metal command queue.
    zsh: segmentation fault
    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840

[C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 31, 2023
Wraps all FlutterTextInputPlugin tests in an autorelease pool to ensure
resources are cleaned up between tests.

Prior to this change, running these tests via:

    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \
        --gtest_repeat=1000 \
	--gtest_filter='FlutterTextInputPluginTest.TestFirstRectForCharacterRange'

Resuling in a test failure:

```
[ RUN      ] FlutterTextInputPluginTest.TestFirstRectForCharacterRange
2023-10-31 10:30:48.497112-0700 flutter_desktop_darwin_unittests[5304:4897209] Could not create Metal command queue.
2023-10-31 10:30:48.497761-0700 flutter_desktop_darwin_unittests[5304:4897209] Unable to create FlutterView; no MTLDevice or MTLCommandQueue available.
2023-10-31 10:30:48.520949-0700 flutter_desktop_darwin_unittests[5304:4897209] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Did not record an invocation in OCMStub/OCMExpect/OCMReject.
Possible causes are:
- The receiver is not a mock object.
- The selector conflicts with a selector implemented by OCMStubRecorder/OCMExpectationRecorder.'
*** First throw call stack:
(
        0   CoreFoundation                      0x000000018b1f6800 __exceptionPreprocess + 176
        1   libobjc.A.dylib                     0x000000018acedeb4 objc_exception_throw + 60
        2   CoreFoundation                      0x000000018b1f66f0 +[NSException exceptionWithName:reason:userInfo:] + 0
        3   flutter_desktop_darwin_unittests    0x00000001039318a0 +[OCMMacroState endStubMacro] + 272
        4   flutter_desktop_darwin_unittests    0x0000000100138858 -[FlutterInputPluginTestObjc testFirstRectForCharacterRange] + 1316
        5   flutter_desktop_darwin_unittests    0x0000000100152e04 _ZN7flutter7testing62FlutterTextInputPluginTest_TestFirstRectForCharacterRange_Test8TestBodyEv + 60
        6   flutter_d
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Did not record an invocation in OCMStub/OCMExpect/OCMReject.
Possible causes are:
- The receiver is not a mock object.
- The selector conflicts with a selector implemented by OCMStubRecorder/OCMExpectationRecorder.'
*** First throw call stack:
(
        0   CoreFoundation                      0x000000018b1f6800 __exceptionPreprocess + 176
        1   libobjc.A.dylib                     0x000000018acedeb4 objc_exception_throw + 60
        2   CoreFoundation                      0x000000018b1f66f0 +[NSException exceptionWithName:reason:userInfo:] + 0
        3   flutter_desktop_darwin_unittests    0x00000001039318a0 +[OCMMacroState endStubMacro] + 272
        4   flutter_desktop_darwin_unittests    0x0000000100138858 -[FlutterInputPluginTestObjc testFirstRectForCharacterRange] + 1316
```

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 31, 2023
The tests for FlutterCompositor are not useful. The current tests test
two things:
1. That the mocks we set up behave the way we set them up to behave.
2. That the implementation of FlutterCompositor is the current
   implementation of FlutterCompositor.

As an example, consider FlutterCompositorTest.TestPresent:
https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L107

Ostensibly, this test verifies that the onPresent callback configured in
our fake FlutterViewProvider implementation is called when
FlutterCompositor::Present() is called.

However, taking a look at the mocking setup:
https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L47-L85

We do the following:
1. Mock out FlutterSurfaceManager such that when a surface is requested,
   we hand back a mock surface. A little gross since we're relying on
   some knowledge of implementation details of the compositor, but let's
   take this as reasonable for now.
2. We mock out `FlutterSurface asFlutterMetalTexture` to return a mock
   texture. Again, we're getting a bit deep into implementation details
   that the test shouldn't know about, but let's assume this gets us
   somewhere.
3. We mock out `FlutterSurfaceManager present:notify:` to actually
   call the `onPresent` callback if it's passed in.

In effect, we're testing that:
1. We configured our mock for `FlutterSurfaceManager present:notify:` to
   call onPresent.
2. That `FlutterCompositor::Present` actually calls
   `FlutterSurfaceManager present:notify:` despite that being a simple
   implementation detail of that call.

This removes these tests. I have filed the following issue to track
refactoring this class for testability and adding tests:
flutter/flutter#137648

Encountered these tests as part of deflaking and cleaning up memory
allocations throughout the macOS desktop tests.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 31, 2023
The tests for FlutterCompositor are not useful. The current tests test
two things:
1. That the mocks we set up behave the way we set them up to behave.
2. That the implementation of FlutterCompositor is the current
   implementation of FlutterCompositor.

As an example, consider FlutterCompositorTest.TestPresent:
https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L107

Ostensibly, this test verifies that the onPresent callback configured in
our fake FlutterViewProvider implementation is called when
FlutterCompositor::Present() is called.

However, taking a look at the mocking setup:
https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L47-L85

We do the following:
1. Mock out FlutterSurfaceManager such that when a surface is requested,
   we hand back a mock surface. A little gross since we're relying on
   some knowledge of implementation details of the compositor, but let's
   take this as reasonable for now.
2. We mock out `FlutterSurface asFlutterMetalTexture` to return a mock
   texture. Again, we're getting a bit deep into implementation details
   that the test shouldn't know about, but let's assume this gets us
   somewhere.
3. We mock out `FlutterSurfaceManager present:notify:` to actually
   call the `onPresent` callback if it's passed in.

In effect, we're testing that:
1. We configured our mock for `FlutterSurfaceManager present:notify:` to
   call onPresent.
2. That `FlutterCompositor::Present` actually calls
   `FlutterSurfaceManager present:notify:` despite that being a simple
   implementation detail of that call.

This removes these tests. I have filed the following issue to track
refactoring this class for testability and adding tests:
flutter/flutter#137648

Encountered these tests as part of deflaking and cleaning up memory
allocations throughout the macOS desktop tests.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Oct 31, 2023
The tests for FlutterCompositor are not useful. The current tests test
two things:
1. That the mocks we set up behave the way we set them up to behave.
2. That the implementation of FlutterCompositor is the current
   implementation of FlutterCompositor.

As an example, consider FlutterCompositorTest.TestPresent:
https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L107

Ostensibly, this test verifies that the onPresent callback configured in
our fake FlutterViewProvider implementation is called when
FlutterCompositor::Present() is called.

However, taking a look at the mocking setup:
https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L47-L85

We do the following:
1. Mock out FlutterSurfaceManager such that when a surface is requested,
   we hand back a mock surface. A little gross since we're relying on
   some knowledge of implementation details of the compositor, but let's
   take this as reasonable for now.
2. We mock out `FlutterSurface asFlutterMetalTexture` to return a mock
   texture. Again, we're getting a bit deep into implementation details
   that the test shouldn't know about, but let's assume this gets us
   somewhere.
3. We mock out `FlutterSurfaceManager present:notify:` to actually
   call the `onPresent` callback if it's passed in.

In effect, we're testing that:
1. We configured our mock for `FlutterSurfaceManager present:notify:` to
   call onPresent.
2. That `FlutterCompositor::Present` actually calls
   `FlutterSurfaceManager present:notify:` despite that being a simple
   implementation detail of that call.

This removes these tests. I have filed the following issue to track
refactoring this class for testability and adding tests:
flutter/flutter#137648

Encountered these tests as part of deflaking and cleaning up memory
allocations throughout the macOS desktop tests.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
auto-submit bot pushed a commit to flutter/engine that referenced this issue Nov 1, 2023
The tests for FlutterCompositor are not useful. The current tests test two things:
1. That the mocks we set up behave the way we set them up to behave.
2. That the implementation of FlutterCompositor is the current implementation of FlutterCompositor.

As an example, consider FlutterCompositorTest.TestPresent: https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L107-L137

Ostensibly, this test verifies that the onPresent callback configured in our fake FlutterViewProvider implementation is called when FlutterCompositor::Present() is called.

However, taking a look at the mocking setup:
https://github.com/flutter/engine/blob/89e8de970cb99aa82e067bbdb4a8e927e53f0b28/shell/platform/darwin/macos/framework/Source/FlutterCompositorTest.mm#L47-L85

We do the following:
1. Mock out FlutterSurfaceManager such that when a surface is requested, we hand back a mock surface. A little gross since we're relying on some knowledge of implementation details of the compositor, but let's take this as reasonable for now.
2. We mock out `FlutterSurface asFlutterMetalTexture` to return a mock texture. Again, we're getting a bit deep into implementation details that the test shouldn't know about, but let's assume this gets us somewhere.
3. We mock out `FlutterSurfaceManager present:notify:` to actually call the `onPresent` callback if it's passed in.

In effect, we're testing that:
1. We configured our mock for `FlutterSurfaceManager present:notify:` to call onPresent.
2. That `FlutterCompositor::Present` actually calls `FlutterSurfaceManager present:notify:` despite that being a simple implementation detail of that call.

This removes these tests. I have filed the following issue to track refactoring this class for testability and adding tests: flutter/flutter#137648

Encountered these tests as part of deflaking and cleaning up memory allocations throughout the macOS desktop tests.

Issue: flutter/flutter#137648
Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840

[C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 8, 2023
Wraps all FlutterViewController tests in an autorelease pool to ensure
resources are cleaned up.

Adds a MockFlutterEngineTest subclass of AutoreleasePoolTest that
creates an OCPartialMock FlutterEngine and shuts it down at the end of
the test. Previously we were not shutting down any FlutterEngine
instances we allocated, resulting in potentially thousands of threads
and graphics contexts being allocated.

Prior to this change, running these tests via:

    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \
        --gtest_filter='FlutterViewController.*' --gtest_repeat=1000

resulted in test failures and sometimes segfaults. This ensures
resources are cleaned up

Eventually all unit tests should configure their FlutterEngine via
either FlutterEngineTest (which should be an AutoreleasePoolTest) or
MockFlutterEngineTest, and the CreateMockFlutterEngine function moved to
a static used in the implementation of these.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 8, 2023
Wraps all FlutterViewController tests in an autorelease pool to ensure
resources are cleaned up.

Adds a MockFlutterEngineTest subclass of AutoreleasePoolTest that
creates an OCPartialMock FlutterEngine and shuts it down at the end of
the test. Previously we were not shutting down any FlutterEngine
instances we allocated, resulting in potentially thousands of threads
and graphics contexts being allocated.

Prior to this change, running these tests via:

    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \
        --gtest_filter='FlutterViewController.*' --gtest_repeat=1000

resulted in test failures and sometimes segfaults. This ensures
resources are cleaned up

Eventually all unit tests should configure their FlutterEngine via
either FlutterEngineTest (which should be an AutoreleasePoolTest) or
MockFlutterEngineTest, and the CreateMockFlutterEngine function moved to
a static used in the implementation of these.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to flutter/engine that referenced this issue Nov 8, 2023
Wraps all FlutterViewController tests in an autorelease pool to ensure
resources are cleaned up.

Adds a MockFlutterEngineTest subclass of AutoreleasePoolTest that
creates an OCPartialMock FlutterEngine and shuts it down at the end of
the test. Previously we were not shutting down any FlutterEngine
instances we allocated, resulting in potentially thousands of threads
and graphics contexts being allocated.

Prior to this change, running these tests via:

    ../out/host_debug_unopt_arm64/flutter_desktop_darwin_unittests \
        --gtest_filter='FlutterViewController.*' --gtest_repeat=1000

resulted in test failures and sometimes segfaults. This ensures
resources are cleaned up

Eventually all unit tests should configure their FlutterEngine via
either FlutterEngineTest (which should be an AutoreleasePoolTest) or
MockFlutterEngineTest, and the CreateMockFlutterEngine function moved to
a static used in the implementation of these.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide] and the [C++,
Objective-C, Java style guides].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I added new tests to check the change I am making or feature I am
adding, or the PR is [test-exempt]. See [testing the engine] for
instructions on writing and running engine tests.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I signed the [CLA].
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#overview
[Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene
[test-exempt]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo
[C++, Objective-C, Java style guides]:
https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
[testing the engine]:
https://github.com/flutter/flutter/wiki/Testing-the-engine
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes
[Discord]: https://github.com/flutter/flutter/wiki/Chat
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 15, 2023
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest,
which ensures all allocations are cleaned up.

Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a
fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of
OCMock in the tests, which has been responsible for flakiness in some
tests, in particular where the mock is used across threads. This test
was not problematic, but the fake makes the tests more readable.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 15, 2023
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest,
which ensures all allocations are cleaned up.

Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a
fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of
OCMock in the tests, which has been responsible for flakiness in some
tests, in particular where the mock is used across threads. This test
was not problematic, but the fake makes the tests more readable.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 15, 2023
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest,
which ensures all allocations are cleaned up.

Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a
fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of
OCMock in the tests, which has been responsible for flakiness in some
tests, in particular where the mock is used across threads. This test
was not problematic, but the fake makes the tests more readable.

Also renames the static keyEvent() function CreateKeyEvent, in line with
our style guide.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 15, 2023
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest,
which ensures all allocations are cleaned up.

Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a
fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of
OCMock in the tests, which has been responsible for flakiness in some
tests, in particular where the mock is used across threads. This test
was not problematic, but the fake makes the tests more readable.

Also renames the static keyEvent() function CreateKeyEvent, in line with
our style guide.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 15, 2023
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest,
which ensures all allocations are cleaned up.

Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a
fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of
OCMock in the tests, which has been responsible for flakiness in some
tests, in particular where the mock is used across threads. This test
was not problematic, but the fake makes the tests more readable.

Also renames the static keyEvent() function CreateKeyEvent, in line with
our style guide.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
auto-submit bot pushed a commit to flutter/engine that referenced this issue Nov 15, 2023
Runs all FlutterChannelKeyResponderTest tests in an AutoReleasepoolTest, which ensures all allocations are cleaned up.

Also replaces the use of OCMock in FlutterChannelKeyResponderTest with a fake FlutterBasicMessageChannel subclass. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840

[C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 15, 2023
Runs all FlutterMenuPlugin tests in an AutoReleasepoolTest, which
ensures all allocations are cleaned up.

Also extracts out some hackery into the FlutterMenuPluginTest fixture to
ensure that NSApplication instantiates everything necessary to do menu
bar manipulation.

Also replaces the use of OCMock in FlutterMenuPluginTest.mm with a fake
FakePluginRegistrar class. This avoids unnecessary use of OCMock in the
tests, which has been responsible for flakiness in some tests, in
particular where the mock is used across threads. This test was not
problematic, but the fake makes the tests more readable.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 15, 2023
Runs all FlutterMenuPlugin tests in an AutoReleasepoolTest, which
ensures all allocations are cleaned up.

Also extracts out some hackery into the FlutterMenuPluginTest fixture to
ensure that NSApplication instantiates everything necessary to do menu
bar manipulation.

Also replaces the use of OCMock in FlutterMenuPluginTest.mm with a fake
FakePluginRegistrar class. This avoids unnecessary use of OCMock in the
tests, which has been responsible for flakiness in some tests, in
particular where the mock is used across threads. This test was not
problematic, but the fake makes the tests more readable.

Also fixes linter warnings about using NSLocalizedString for user-facing
strings.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to flutter/engine that referenced this issue Nov 16, 2023
Runs all FlutterMenuPlugin tests in an AutoReleasepoolTest, which                   
ensures all allocations are cleaned up.                                             
                                                                                    
Also extracts out some hackery into the FlutterMenuPluginTest fixture to            
ensure that NSApplication instantiates everything necessary to do menu              
bar manipulation.                                                                   
                                                                                    
Also replaces the use of OCMock in FlutterMenuPluginTest.mm with a fake             
FakePluginRegistrar class. This avoids unnecessary use of OCMock in the             
tests, which has been responsible for flakiness in some tests, in                   
particular where the mock is used across threads. This test was not                 
problematic, but the fake makes the tests more readable.                            
                                                                                    
Also fixes linter warnings about using NSLocalizedString for user-facing            
strings.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 16, 2023
Also replaces the use of OCMock with a fake for tests involving the use
of NSPasteboard. This avoids unnecessary use of OCMock in the tests,
which has been responsible for flakiness in some tests, in particular
where the mock is used across threads. This test was not problematic,
but the fake makes the tests more readable.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 16, 2023
Also replaces the use of OCMock with a fake for tests involving the use
of NSPasteboard. This avoids unnecessary use of OCMock in the tests,
which has been responsible for flakiness in some tests, in particular
where the mock is used across threads. This test was not problematic,
but the fake makes the tests more readable.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 16, 2023
Also replaces the use of OCMock with a fake for tests involving the use
of NSPasteboard. This avoids unnecessary use of OCMock in the tests,
which has been responsible for flakiness in some tests, in particular
where the mock is used across threads. This test was not problematic,
but the fake makes the tests more readable.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
cbracken added a commit to cbracken/flutter_engine that referenced this issue Nov 16, 2023
Also replaces the use of OCMock with a fake for tests involving the use
of NSPasteboard. This avoids unnecessary use of OCMock in the tests,
which has been responsible for flakiness in some tests, in particular
where the mock is used across threads. This test was not problematic,
but the fake makes the tests more readable.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840
auto-submit bot pushed a commit to flutter/engine that referenced this issue Nov 16, 2023
Also replaces the use of OCMock with a fake for tests involving the use of NSPasteboard. This avoids unnecessary use of OCMock in the tests, which has been responsible for flakiness in some tests, in particular where the mock is used across threads. This test was not problematic, but the fake makes the tests more readable.

Issue: flutter/flutter#104789
Issue: flutter/flutter#127441
Issue: flutter/flutter#124840

[C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
@cbracken cbracken self-assigned this Nov 16, 2023
@cbracken cbracken changed the title Make tests in flutter_desktop_darwin_unittests self contained Deflake flutter_desktop_darwin_unittests Nov 16, 2023
@cbracken
Copy link
Member

Renamed to capture the broader issue of deflaking the macOS desktop tests, and as part of that work, eliminate the use of OCMock.

@flutter-triage-bot flutter-triage-bot bot added the Bot is counting down the days until it unassigns the issue label Mar 21, 2024
@flutter-triage-bot
Copy link

This issue is assigned to @cbracken but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!

@cbracken cbracken removed their assignment Mar 29, 2024
@flutter-triage-bot flutter-triage-bot bot removed the Bot is counting down the days until it unassigns the issue label Mar 29, 2024
@cbracken
Copy link
Member

cbracken commented Apr 10, 2024

I'm going to close this since we've made a ton of progress on this and I don't think we're seeing a significant number of flakes on the bots anymore. The underlying issue of dependencies between tests still exists for some tests, but not more than on any other platform.

Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
engine flutter/engine repository. See also e: labels. P2 Important issues not at the top of the work list platform-mac Building on or for macOS specifically team-engine Owned by Engine team triaged-engine Triaged by Engine team
Projects
None yet
Development

No branches or pull requests

3 participants