Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Mac_ios native_platform_view_ui_tests_ios is 1.01% flaky #109697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fluttergithubbot opened this issue Aug 17, 2022 · 6 comments · Fixed by #109720 or #111139
Closed

Mac_ios native_platform_view_ui_tests_ios is 1.01% flaky #109697

fluttergithubbot opened this issue Aug 17, 2022 · 6 comments · Fixed by #109720 or #111139
Assignees
@hellohuanlin
Labels
c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression

Comments

@fluttergithubbot
Copy link
Contributor

The post-submit test builder Mac_ios native_platform_view_ui_tests_ios, which has been marked bringup: true, had 1 flakes over past 99 commits.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_ios%20native_platform_view_ui_tests_ios/224
Commit: 20ffb32

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_ios%20native_platform_view_ui_tests_ios/224

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_ios%20native_platform_view_ui_tests_ios

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).
@fluttergithubbot fluttergithubbot added P1 c: flake Tests that sometimes, but not always, incorrectly pass labels Aug 17, 2022
@hellohuanlin
Copy link
Contributor

hellohuanlin commented Aug 17, 2022 •
edited
Loading

Last time it was 8% flaky, so looks like the longer wait time indeed solved the previous issue with element not showing up yet.

I suspect that this is just one time failure, possibly due to failure installing the app or removing the previously installed app. (at 0.07 it tries to terminate the app with the same bundle ID) The error indicates that the app is not even launched yet:

[native_platform_view_ui_tests_ios] [STDOUT] stdout: 2022-08-15 04:07:00.782923-0700 PlatformViewUITests-Runner[1033:33791] Running tests...
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'All tests' started at 2022-08-15 04:07:01.287
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'PlatformViewUITests.xctest' started at 2022-08-15 04:07:01.289
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'PlatformViewUITests' started at 2022-08-15 04:07:01.290
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Case '-[PlatformViewUITests testPlatformViewFocus]' started.
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.00s Start Test at 2022-08-15 04:07:01.292
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.03s Set Up
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.04s     Open io.flutter.platformview.platformviewTests
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.07s         Launch io.flutter.platformview.platformviewTests
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.07s             Terminate io.flutter.platformview.platformviewTests:265
[native_platform_view_ui_tests_ios] [STDOUT] stdout: /opt/s/w/ir/x/w/recipe_cleanup/tmpfedtih2f/flutter sdk/dev/integration_tests/ios_platform_view_tests/ios/PlatformViewUITests/PlatformViewUITests.m:29: error: -[PlatformViewUITests testPlatformViewFocus] : Failed to terminate io.flutter.platformview.platformviewTests:265: Failed to terminate io.flutter.platformview.platformviewTests:0
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =    60.19s Tear Down
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Case '-[PlatformViewUITests testPlatformViewFocus]' failed (60.192 seconds).
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'PlatformViewUITests' failed at 2022-08-15 04:08:01.484.
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 	 Executed 1 test, with 1 failure (0 unexpected) in 60.192 (60.194) seconds
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'PlatformViewUITests.xctest' failed at 2022-08-15 04:08:01.485.
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 	 Executed 1 test, with 1 failure (0 unexpected) in 60.192 (60.196) seconds
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test Suite 'All tests' failed at 2022-08-15 04:08:01.486.
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 	 Executed 1 test, with 1 failure (0 unexpected) in 60.192 (60.199) seconds
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 2022-08-15 04:08:01.440 xcodebuild[61456:1220877] [MT] IDETestOperationsObserverDebug: 64.322 elapsed -- Testing started completed.
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 2022-08-15 04:08:01.440 xcodebuild[61456:1220877] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 2022-08-15 04:08:01.440 xcodebuild[61456:1220877] [MT] IDETestOperationsObserverDebug: 64.322 sec, +64.322 sec -- end
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 
[native_platform_view_ui_tests_ios] [STDOUT] stdout: Test session results, code coverage, and logs:
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 	/opt/s/w/ir/x/t/flutter_xcresult.pgxJBd/result
[native_platform_view_ui_tests_ios] [STDOUT] stdout: 
[native_platform_view_ui_tests_ios] [STDOUT] stderr: Failing tests:
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 	PlatformViewUITests:
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 		-[PlatformViewUITests testPlatformViewFocus]
[native_platform_view_ui_tests_ios] [STDOUT] stderr: 
[native_platform_view_ui_tests_ios] [STDOUT] stderr: ** TEST FAILED **

I will close this issue for now and see if it reappears. What do you think @jmagman

@jmagman
Copy link
Member

jmagman commented Aug 17, 2022

@keyonghan If the flaky threshold is under 2% shouldn't the bot be creating a PR to mark this no longer flaky, as opposed to a new issue?

@keyonghan
Copy link
Contributor

We will stop creating new bugs for already bringup:true builders: #107563

The logic to deflake a target is no single flake happens in recent 50 runs.

Taking look at the history, seems there are still 6 failures/flakes in recent 100 commits:
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_ios%20native_platform_view_ui_tests_ios/233/overview
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_ios%20native_platform_view_ui_tests_ios/226/overview
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_ios%20native_platform_view_ui_tests_ios/224/overview
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_ios%20native_platform_view_ui_tests_ios/194/overview
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_ios%20native_platform_view_ui_tests_ios/183/overview
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_ios%20native_platform_view_ui_tests_ios/173/overview

@hellohuanlin
Copy link
Contributor

interesting, not sure why it's reported 1% only.

Looking into the failures, seems all of them are the same issue with install:

Test Case '-[PlatformViewUITests testPlatformViewFocus]' started.
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.00s Start Test at 2022-08-11 20:45:54.319
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.03s Set Up
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.04s     Open io.flutter.platformview.platformviewTests
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.07s         Launch io.flutter.platformview.platformviewTests
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =     0.07s             Terminate io.flutter.platformview.platformviewTests:257
[native_platform_view_ui_tests_ios] [STDOUT] stdout: /opt/s/w/ir/x/w/recipe_cleanup/tmp_l_a81ax/flutter sdk/dev/integration_tests/ios_platform_view_tests/ios/PlatformViewUITests/PlatformViewUITests.m:29: error: -[PlatformViewUITests testPlatformViewFocus] : Failed to terminate io.flutter.platformview.platformviewTests:257: Failed to terminate io.flutter.platformview.platformviewTests:0
[native_platform_view_ui_tests_ios] [STDOUT] stdout:     t =    60.20s Tear Down

This is different from the previous failure which should have been fixed by increasing the timeout. I will take a further look into this.

@jmagman
Copy link
Member

jmagman commented Aug 17, 2022

Yeah weird the app isn't even launching, the screenshots only show the home screen. Every failure is on a different devicelab host so it doesn't obviously look environment specific.

@hellohuanlin hellohuanlin mentioned this issue Aug 17, 2022
Merged
8 tasks
This was referenced Aug 22, 2022
Merged
Merged
@github-actions
Copy link

github-actions bot commented Sep 1, 2022

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 1, 2022
@flutter-triage-bot flutter-triage-bot bot added P0 Critical issues such as a build break or regression and removed P1 labels Jun 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@hellohuanlin hellohuanlin

Labels
c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression
Projects
None yet
Milestone
No milestone
4 participants
@jmagman @hellohuanlin @fluttergithubbot @keyonghan