[platform_view]add timeout for alert buttons and a few other changes to fix a test flake #110516

hellohuanlin · 2022-08-29T17:30:14Z

This is to add timeout for alert buttons, to address a new test flake.

here's led

Analysis

1. `kAXErrorServerNotFound`:

A similar issue is here, which is solved by calling terminate in tearDown. However, this is what we are doing already.
Another issue is here, which seems to be resolved by bumping xcode version
Maybe related to this stackoverflow question. But the top voted answer is what we are doing already.
I suspect that kAXErrorServerNotFound is just a warning and not an error, because the test continues to run even after this error, and then we got another error about alert not found.

2. "No matches found for Descendants matching type Alert"

Here's a list of many ways of deleting the app from springboard. They are pretty much the same. Some answers wait for the alert, and others do not.
When I tested it on my machine, I did not need to explicitly wait for the alert and it worked.
But since it's complaining about alert not found, I think it's our best bet to wait for the alert.

Changes

1. Add the timeout for the alert

As discussed in above. Also added timeout for the initial delete button just to be extra safe.

2. Use context menu's delete button instead of the "X" button

In this stackoverflow question, some answers use the X button, and others use the context menu's button
The "X" button API is undocumented, some answers use @"DeleteButton" name, and others use a hardcoded coordinate offset. Both aren't ideal. Very interestingly, I tried the @"DeleteButton" approach again on my machine today, and some times it does not work (but most of the time it works)
For the context menu approach, some solutions uses 1.3 seconds, and others use 2 seconds. I think 2 seconds is a bit safer in case of slow hardware.

3. Add a hittable check for appIcon

It doesn't appear in any of the answers in that stackoverflow question, but some commented out about the app icon not being hittable.

Debug info

Full logs

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8804511299285366417/+/u/run_native_platform_view_ui_tests_ios/test_stdout

Logs


[native_platform_view_ui_tests_ios] [STDOUT] stdout: 2022-08-29 02:08:40.593596-0700 PlatformViewUITests-Runner[1932:59533] Running tests...
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'All tests' started at 2022-08-29 02:08:41.107
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'PlatformViewUITests.xctest' started at 2022-08-29 02:08:41.109
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'PlatformViewUITests' started at 2022-08-29 02:08:41.110
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Case '-[PlatformViewUITests testPlatformViewFocus]' started.
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.00s Start Test at 2022-08-29 02:08:41.113
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.03s Set Up
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.05s     Open com.apple.springboard
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.08s         Activate com.apple.springboard
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.09s             Wait for com.apple.springboard to idle
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.13s     Waiting 60.0s for "ios_platform_view_tests" Icon to exist
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.14s         Checking `Expect predicate `exists == 1` for object "ios_platform_view_tests" Icon`
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.14s             Checking existence of `"ios_platform_view_tests" Icon`
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.25s                 Ignoring failure to get hierarchy for remote element in process 1396 (Error getting main window kAXErrorServerNotFound)
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 2022-08-29 02:08:42.377445-0700 PlatformViewUITests-Runner[1932:59533] Deleting previously installed app.
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.27s     Press "ios_platform_view_tests" Icon for 3.0 seconds
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.27s         Wait for com.apple.springboard to idle
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.30s         Find the "ios_platform_view_tests" Icon
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.37s             Ignoring failure to get hierarchy for remote element in process 1396 (Error getting main window kAXErrorServerNotFound)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.39s         Check for interrupting elements affecting "ios_platform_view_tests" Icon
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.46s             Ignoring failure to get hierarchy for remote element in process 1396 (Error getting main window kAXErrorServerNotFound)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     1.47s         Synthesize event
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     3.64s         Wait for com.apple.springboard to idle
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     4.04s     Tap "DeleteButton" Button
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     4.04s         Wait for com.apple.springboard to idle
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     4.09s         Find the "DeleteButton" Button
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     4.25s             Ignoring failure to get hierarchy for remote element in process 1396 (Error getting main window kAXErrorServerNotFound)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     5.26s             Find the "DeleteButton" Button (retry 1)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     5.42s                 Ignoring failure to get hierarchy for remote element in process 1396 (Error getting main window kAXErrorServerNotFound)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     5.44s         Check for interrupting elements affecting "DeleteButton" Button
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     5.53s             Ignoring failure to get hierarchy for remote element in process 1396 (Error getting main window kAXErrorServerNotFound)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     5.54s         Synthesize event
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     5.78s         Wait for com.apple.springboard to idle
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     6.14s     Tap "Delete App" Button
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     6.14s         Wait for com.apple.springboard to idle
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     6.17s         Find the "Delete App" Button
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     6.25s             Ignoring failure to get hierarchy for remote element in process 1396 (Error getting main window kAXErrorServerNotFound)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     7.26s             Find the "Delete App" Button (retry 1)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     7.36s                 Ignoring failure to get hierarchy for remote element in process 1396 (Error getting main window kAXErrorServerNotFound)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     8.37s             Find the "Delete App" Button (retry 2)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     8.47s                 Ignoring failure to get hierarchy for remote element in process 1396 (Error getting main window kAXErrorServerNotFound)
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     8.48s             Collecting extra data to assist test failure triage
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     8.48s                 Requesting snapshot of accessibility hierarchy for app with pid 62
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     8.58s                 Requesting snapshot of accessibility hierarchy for app with pid 62
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     8.68s                 Find: Descendants matching type Alert
[native_platform_view_ui_tests_ios] [STDOUT] stdout: /opt/s/w/ir/x/w/recipe_cleanup/tmpf89bzsws/flutter sdk/dev/integration_tests/ios_platform_view_tests/ios/PlatformViewUITests/PlatformViewUITests.m:45: error: -[PlatformViewUITests testPlatformViewFocus] : Failed to get matching snapshot: No matches found for Descendants matching type Alert from input {(
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     Application, pid: 62, label: ' '
[native_platform_view_ui_tests_ios] [STDOUT] stdout: )}
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     8.78s Tear Down
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Case '-[PlatformViewUITests testPlatformViewFocus]' failed (8.781 seconds).
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'PlatformViewUITests' failed at 2022-08-29 02:08:49.892.
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 	 Executed 1 test, with 1 failure (0 unexpected) in 8.781 (8.783) seconds
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'PlatformViewUITests.xctest' failed at 2022-08-29 02:08:49.893.
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 	 Executed 1 test, with 1 failure (0 unexpected) in 8.781 (8.784) seconds
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'All tests' failed at 2022-08-29 02:08:49.894.
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 	 Executed 1 test, with 1 failure (0 unexpected) in 8.781 (8.787) seconds
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 2022-08-29 02:08:49.991 xcodebuild[56698:249181] [MT] IDETestOperationsObserverDebug: 12.427 elapsed -- Testing started completed.
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 2022-08-29 02:08:49.991 xcodebuild[56698:249181] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 2022-08-29 02:08:49.991 xcodebuild[56698:249181] [MT] IDETestOperationsObserverDebug: 12.427 sec, +12.427 sec -- end
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test session results, code coverage, and logs:
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 	/opt/s/w/ir/x/t/flutter_xcresult.GkWOcN/result
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 
[native_platform_view_ui_tests_ios] [STDOUT] stderr: Failing tests:
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 	PlatformViewUITests:
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 		-[PlatformViewUITests testPlatformViewFocus]
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 
[native_platform_view_ui_tests_ios] [STDOUT] stderr: ** TEST FAILED **

List which issues are fixed by this PR. You must list at least one issue.
#109697

If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is test-exempt.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

jmagman · 2022-08-29T18:13:18Z

dev/integration_tests/ios_platform_view_tests/ios/PlatformViewUITests/PlatformViewUITests.m

@@ -36,17 +36,31 @@ - (void)setUp {
  if ([appIcon waitForExistenceWithTimeout:kStandardTimeOut]) {
    NSLog(@"Deleting previously installed app.");

-    // Make icons wiggle
-    [appIcon pressForDuration:3];
+    if (!appIcon.isHittable) {


Instead of sleeping, use the wait APIs

flutter/dev/integration_tests/ios_host_app/FlutterUITests/FlutterUITests.m

Lines 120 to 122 in 7428ab6

NSPredicate *hittable = [NSPredicate predicateWithFormat:@"exists == YES AND hittable == YES"];

[self expectationForPredicate:hittable evaluatedWithObject:element handler:nil];

[self waitForExpectationsWithTimeout:30.0 handler:nil];

jmagman · 2022-08-29T18:21:56Z

dev/integration_tests/ios_platform_view_tests/ios/PlatformViewUITests/PlatformViewUITests.m

+    // Pressing for 2 seconds will bring up context menu.
+    // Pressing for 3 seconds will dismiss the context menu and make icons wiggle.
+    [appIcon pressForDuration:2];


I hope this isn't flaky... You could consider taking XCAttachment screenshots when these fail to see what's going on.
I hate these kinds of Xcode test harness bugs. As do you too now!

Yeah I was thinking the same. In that stackoverflow I linked, there are 2 versions: 1.3 seconds and 2 seconds.

I picked 2 seconds in case of slow hardware.

Added screenshot

Hm actually sorry to thrash on this, but never mind this isn't necessary, the next thing that happens is the screenshotAttachment XCTAssert that will automatically take a screenshot if the assert fails.

… when assert fails

…changes to fix a test flake (flutter/flutter#110516)

flutter-dashboard bot added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Aug 29, 2022

[platform_view]add timeout for alert element

002c4cf

hellohuanlin force-pushed the platform_view_alert_timeout branch from 41b3674 to 002c4cf Compare August 29, 2022 17:48

hellohuanlin requested review from jmagman and cyanglaz August 29, 2022 18:06

hellohuanlin changed the title ~~[platform_view]add timeout for alert buttons~~ [platform_view]add timeout for alert buttons and a few other changes Aug 29, 2022

hellohuanlin changed the title ~~[platform_view]add timeout for alert buttons and a few other changes~~ [platform_view]add timeout for alert buttons and a few other changes to fix a test flake Aug 29, 2022

jmagman reviewed Aug 29, 2022

View reviewed changes

use waitForExpectation to check hittable

191683c

hellohuanlin requested a review from jmagman August 29, 2022 18:18

jmagman reviewed Aug 29, 2022

View reviewed changes

add screenshot

2c779eb

hellohuanlin requested a review from jmagman August 29, 2022 18:32

remove duplicate screenshot attachment since xcode auto screenshot it…

aa81312

… when assert fails

jmagman approved these changes Aug 29, 2022

View reviewed changes

hellohuanlin added the autosubmit Merge PR when tree becomes green via auto submit App label Aug 30, 2022

auto-submit bot merged commit ee98003 into flutter:master Aug 30, 2022

This was referenced Aug 30, 2022

Roll Flutter from f66a09791920 to dfcd9650e36b (26 revisions) flutter/packages#2529

Closed

Roll Flutter from 5848306e3784 to dfcd9650e36b (24 revisions) flutter/plugins#6336

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Aug 30, 2022

ee98003 [platform_view]add timeout for alert buttons and a few other …

c1024aa

…changes to fix a test flake (flutter/flutter#110516)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Aug 30, 2022

ee98003 [platform_view]add timeout for alert buttons and a few other …

fb4148c

…changes to fix a test flake (flutter/flutter#110516)

engine-flutter-autoroll mentioned this pull request Aug 30, 2022

Roll Flutter from f66a09791920 to 2ed99fb70d08 (27 revisions) flutter/packages#2530

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Aug 30, 2022

ee98003 [platform_view]add timeout for alert buttons and a few other …

50f53af

…changes to fix a test flake (flutter/flutter#110516)

engine-flutter-autoroll mentioned this pull request Sep 1, 2022

Roll Flutter from f66a09791920 to 759d0e1b20c1 (56 revisions) flutter/packages#2541

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Sep 1, 2022

ee98003 [platform_view]add timeout for alert buttons and a few other …

b778c4a

…changes to fix a test flake (flutter/flutter#110516)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[platform_view]add timeout for alert buttons and a few other changes to fix a test flake #110516

[platform_view]add timeout for alert buttons and a few other changes to fix a test flake #110516

Uh oh!

hellohuanlin commented Aug 29, 2022 •

edited

Loading

Uh oh!

jmagman Aug 29, 2022

Uh oh!

hellohuanlin Aug 29, 2022

Uh oh!

jmagman Aug 29, 2022

Uh oh!

hellohuanlin Aug 29, 2022

Uh oh!

hellohuanlin Aug 29, 2022 •

edited

Loading

Uh oh!

jmagman Aug 29, 2022

Uh oh!

Uh oh!

	NSPredicate *hittable = [NSPredicate predicateWithFormat:@"exists == YES AND hittable == YES"];
	[self expectationForPredicate:hittable evaluatedWithObject:element handler:nil];
	[self waitForExpectationsWithTimeout:30.0 handler:nil];

[platform_view]add timeout for alert buttons and a few other changes to fix a test flake #110516

[platform_view]add timeout for alert buttons and a few other changes to fix a test flake #110516

Uh oh!

Conversation

hellohuanlin commented Aug 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Analysis

1. kAXErrorServerNotFound:

2. "No matches found for Descendants matching type Alert"

Changes

1. Add the timeout for the alert

2. Use context menu's delete button instead of the "X" button

3. Add a hittable check for appIcon

Debug info

Full logs

Logs

Pre-launch Checklist

Uh oh!

jmagman Aug 29, 2022

Choose a reason for hiding this comment

Uh oh!

hellohuanlin Aug 29, 2022

Choose a reason for hiding this comment

Uh oh!

jmagman Aug 29, 2022

Choose a reason for hiding this comment

Uh oh!

hellohuanlin Aug 29, 2022

Choose a reason for hiding this comment

Uh oh!

hellohuanlin Aug 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmagman Aug 29, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hellohuanlin commented Aug 29, 2022 •

edited

Loading

1. `kAXErrorServerNotFound`:

hellohuanlin Aug 29, 2022 •

edited

Loading