Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Linux_android complex_layout_semantics_perf is 2.06% flaky #108888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fluttergithubbot opened this issue Aug 3, 2022 · 3 comments · Fixed by #108906
Closed

Linux_android complex_layout_semantics_perf is 2.06% flaky #108888

fluttergithubbot opened this issue Aug 3, 2022 · 3 comments · Fixed by #108906
Assignees
@goderbauer
Labels
c: flake Tests that sometimes, but not always, incorrectly pass engine flutter/engine repository. See also e: labels. P0 Critical issues such as a build break or regression

Comments

@fluttergithubbot
Copy link
Contributor

The post-submit test builder Linux_android complex_layout_semantics_perf had a flaky ratio 2.06% for the past (up to) 100 commits, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux_android%20complex_layout_semantics_perf/4066
Commit: 4304411

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux_android%20complex_layout_semantics_perf/4066
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux_android%20complex_layout_semantics_perf/4064

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux_android%20complex_layout_semantics_perf

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).
@fluttergithubbot fluttergithubbot added engine flutter/engine repository. See also e: labels. P1 c: flake Tests that sometimes, but not always, incorrectly pass labels Aug 3, 2022
@fluttergithubbot fluttergithubbot mentioned this issue Aug 3, 2022
Closed
@zanderso
Copy link
Member

zanderso commented Aug 3, 2022

In both cases the failure is:

[complex_layout_semantics_perf] [STDOUT] stdout: [  +31 ms] 00:02 �[32m+0�[0m�[31m -1�[0m: semantics performance test initial tree creation �[1m�[31m[E]�[0m�[0m
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   Expected: true
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]     Actual: <false>
[complex_layout_semantics_perf] [STDOUT] stdout: [  +22 ms]   package:test_api/src/expect/expect.dart 134:31                 fail
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   package:test_api/src/expect/expect.dart 129:3                  _expect
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   package:test_api/src/expect/expect.dart 46:3                   expect
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   test_driver/semantics_perf_test.dart 34:9                      main.<fn>.<fn>.<fn>
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   ===== asynchronous gap ===========================
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   package:flutter_driver/src/driver/vmservice_driver.dart 467:5  VMServiceFlutterDriver.traceAction
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   ===== asynchronous gap ===========================
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   test_driver/semantics_perf_test.dart 33:33                     main.<fn>.<fn>
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   ===== asynchronous gap ===========================
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   package:test_api/src/backend/declarer.dart 215:9               Declarer.test.<fn>.<fn>
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   ===== asynchronous gap ===========================
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   package:test_api/src/backend/declarer.dart 213:7               Declarer.test.<fn>
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   ===== asynchronous gap ===========================
[complex_layout_semantics_perf] [STDOUT] stdout: [        ]   package:test_api/src/backend/invoker.dart 257:7                Invoker._waitForOutstandingCallbacks.<fn>
[complex_layout_semantics_perf] [STDOUT] stdout: [        ] 00:02 �[32m+0�[0m�[31m -1�[0m: semantics performance test (tearDownAll)�[0m

Routing to @goderbauer. Also cc @jacob314 who increased a delay that could be related to flakiness: https://github.com/flutter/flutter/blob/master/dev/benchmarks/complex_layout/test_driver/semantics_perf_test.dart#L29

@zanderso zanderso assigned goderbauer and unassigned zanderso Aug 3, 2022
@goderbauer
Copy link
Member

The stack trace is pointing to a failure at test_driver/semantics_perf_test.dart 34:9, which is the expect in this line:

flutter/dev/benchmarks/complex_layout/test_driver/semantics_perf_test.dart

Lines 33 to 35 in 5d52d31

final Timeline timeline = await driver.traceAction(() async {
expect(await driver.setSemantics(true), isTrue);
});

From the failure message it sounds like await driver.setSemantics(true) actually resolves to false when we expect this to evaluate to true.

setSemantics returns "true when the call actually changed the state from on to off or vice versa." [1]

If setSemantics(true) returns false that means semantics were already enabled, which is not what the test expects since it wants to measure the cost of generating the initial semantics tree. Semantics could be enabled unexpectedly because an accessibility service was running on these devices.

@godofredoc Does the test harness relay at all on an accessibility services to control the device to bring up the test? In that case, we need to wait until those services are turned off again before actually running this test.

Another possibility is that a previous test is not hermetic and turns on accessibility services without properly turning it off again at the end. android_semantics_testing [2] is an example of a test that turns on accessibility services. Maybe it (or a similar test) doesn't turn it off properly and reliably at the end? Maybe the test harness should enforce that there are no leaks and force a11y to off between tests?

In the meantime, I'll prepare a PR that forces a11y services to off at the beginning of this test to see if that fixes the flakiness here.

[1] https://master-api.flutter.dev/flutter/flutter_driver/FlutterDriver/setSemantics.html
[2]

flutter/dev/integration_tests/android_semantics_testing/test_driver/main_test.dart

Line 85 in dbadee0

final io.Process run = await io.Process.start(adbPath(), const <String>[

@goderbauer goderbauer mentioned this issue Aug 3, 2022
Merged
@github-actions
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 17, 2022
@flutter-triage-bot flutter-triage-bot bot added P0 Critical issues such as a build break or regression and removed P1 labels Jun 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@goderbauer goderbauer

Labels
c: flake Tests that sometimes, but not always, incorrectly pass engine flutter/engine repository. See also e: labels. P0 Critical issues such as a build break or regression
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Force a11y services to off for complex_layout_semantics_perf test goderbauer/flutter
3 participants
@goderbauer @zanderso @fluttergithubbot