Add flag to disable automatic device reboot for DeviceLab tests #164184

harri35 · 2025-02-26T13:01:19Z

NOTE: @ergosarapu is the author of the pull request and the changes. I just ended up creating it so I could give some early feedback.

Pre-launch Checklist

I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
I read the [Tree Hygiene] wiki page, which explains my responsibilities.
I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
I signed the [CLA].
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is [test-exempt].
I followed the [breaking change policy] and added [Data Driven Fixes] where supported.
All existing and new tests are passing.

google-cla · 2025-02-26T13:01:24Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

harri35 · 2025-02-26T13:32:50Z

dev/devicelab/bin/run.dart

@@ -75,6 +75,9 @@ Future<void> main(List<String> rawArgs) async {
  /// Use an emulator for this test if it is an android test.
  final bool useEmulator = (args['use-emulator'] as bool?) ?? false;

+  /// Do not attempt to reboot device.


Maybe it would be valuable to emphasise the nature of the reboot - that it is a regular automatic reboot based on total amount of tests run on this specific device.

Suggested change

/// Do not attempt to reboot device.

/// Do not automatically reboot the device after a certain number of tests.

/// In LUCI, every device is automatically restarted after a set number of tests.

/// Disabling the automatic reboot can be useful when running tests locally in a loop.

Good suggestion, included

harri35 · 2025-02-26T13:39:16Z

dev/devicelab/bin/run.dart

@@ -75,6 +75,9 @@ Future<void> main(List<String> rawArgs) async {
  /// Use an emulator for this test if it is an android test.
  final bool useEmulator = (args['use-emulator'] as bool?) ?? false;

+  /// Do not attempt to reboot device.
+  final bool noReboot = (args['no-reboot'] as bool?) ?? false;


Just a nitpick - maybe the command line flag name can be improved for something even more descriptive?

"disable-reboot"

"disable-automatic-reboot"

Good suggestion, using disable-reboot now

harri35 · 2025-02-26T13:42:26Z

dev/devicelab/bin/run.dart

+      'no-reboot',
+      negatable: false,
+      help:
+          'Do not attempt to reboot the device after every 30 test runs',


I like the help message. Just wondering if we should write the 30 in here or try to avoid it and say something like

Suggested change

'Do not attempt to reboot the device after every 30 test runs',

'Do not automatically reboot the device after a certain number of tests',

Good suggestion

harri35 · 2025-02-26T13:53:53Z

dev/devicelab/lib/framework/framework.dart

@@ -238,7 +241,9 @@ class _TaskRunner {
      print(stackTrace);
      return TaskResult.failure('Task timed out after $taskTimeout');
    } finally {
-      await checkForRebootRequired();
+      if (!noReboot){


The double negative here can be hard to read IMHO. Maybe flip the variable and use something like:

Suggested change

if (!noReboot){

if (!rebootAutomatically){

The negating is influenced by existing default logic, which cannot be changed (reboot by default). Removing the negating here means the negation will be some level up in the code call stack, so it doesn't really improve the readablity. Keeping it as it is.

harri35 · 2025-02-26T13:59:17Z

Let's also update the DeviceLab Readme at dev/devicelab/README.md and

Add the info about the flag
If possible, add a section for running 0/2000 fail tests locally. Maybe also integrate the tips about configuring the laptop and the device:

Device should be prepped
- Connected using a fast connection (skip USB hubs if not sure about their speed)
- The device should have developer options enabled
- The device should have kept awake (Keep Awake from developer settings)
- Play Store APK scanning should be disabled (blocks automatic install at a random time)
- Device theft protection should be disabled (can auto-lock the device in case of connection loss or if you use Aeroplane mode)
- Google Assistant or Gemini voice detection or similar should be disabled
Computer should be prepped
- The computer user to run tests should be kept awake (Mac caffeinated, Windows PowerToys Awake, ..)
Useful commands on Mac / Linux
- grep -wo -F -i "Starting process for task" log.txt | wc -l
Useful commands on Windows (PowerShell 7)
- Select-String -Path log.txt -Pattern "Starting process for task" | Measure-Object | Select-Object -ExpandProperty Count

harri35 · 2025-02-27T10:45:06Z

One additional note (no action needed) - I personally do not like the naming of the "-exit" flag. It is misleading and hard to understand.
But because it is already there (as a public API interface) then I guess changing it would be difficult. We would have to make an alias and depricate this one and then remove it sometime later. So I would not go into this.

harri35 · 2025-02-27T11:14:34Z

Also did a quick test - works as expected ✅

hyperfine -r 34 --style=full 'dart bin/test_runner.dart test -t integration_ui_keyboard_resize --exit --no-reboot > log.txt' runs until finished
hyperfine -r 34 --style=full 'dart bin/test_runner.dart test -t integration_ui_keyboard_resize --exit > log.txt' stops with a reboot

flutter-dashboard · 2025-03-03T17:59:31Z

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption, contact "@test-exemption-reviewer" in the #hackers channel in Discord (don't just cc them here, they won't see it!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. The test exemption team is a small volunteer group, so all reviewers should feel empowered to ask for tests, without delegating that responsibility entirely to the test exemption group.

Piinks · 2025-06-04T23:01:54Z

Hey @harri35, do you want to continue working on this PR?

harri35 · 2025-06-05T10:14:03Z

Hei @Piinks!

At the time of creation, the change here seemed like a good idea and something that would be needed going forward. Now I am not sure anymore :) Is this valuable for the community?
Or will tests be rearranged in a way that makes the use case described in #164486 not relevant? If it is the latter, then I would close this PR. No point in adding functionality and making things more complicated if no one will use it.

As this was originally Ergo's contribution, I'll also ask Ergo's opinion and get back to you.

Sorry about leaving this hanging here like this. I needed to jump on another thing, and then promptly forgot about this.

Piinks · 2025-06-11T20:08:58Z

Thanks for the explanation! I will defer to @matanlurey if this is an infra change we want to make.

ergosarapu · 2025-06-16T08:18:22Z

The need for this PR raised from the assumption that we can work with flaky tests locally by running test repeatedly hundreds or thousand of times. IIRC we were able to trigger flakiness with this method in 1 or 2 cases only and still not reproducible all the time. Most (if not all) flakiness was related to running tests in the CI infra, rendering the approach of working with flakes actually useless. Therefore I think this change is not needed any more - there are simply no consumers of the functionality.

harri35 commented Feb 26, 2025

View reviewed changes

ergosarapu force-pushed the feature/test-repetition branch 2 times, most recently from f28c10e to ffa717f Compare March 3, 2025 15:24

ergosarapu added 3 commits March 3, 2025 17:25

Add flag to disable attempting device reboot

b8deeae

Rename no-reboot to disable-reboot

c6384bf

Add instructions to run tests repeatedly without device reboot

609c72a

ergosarapu force-pushed the feature/test-repetition branch from ffa717f to 609c72a Compare March 3, 2025 15:25

harri35 assigned harri35 and unassigned harri35 Mar 3, 2025

ergosarapu mentioned this pull request Mar 3, 2025

Allow running tests repeatedly to reproduce flakiness #164486

Open

harri35 marked this pull request as ready for review March 3, 2025 17:57

harri35 requested a review from matanlurey as a code owner March 3, 2025 17:57

harri35 added infra: device lab Infrastructure device lab problems fyi-infra For the attention of Infrastructure team labels Mar 3, 2025

harri35 assigned harri35 and unassigned harri35 Mar 10, 2025

matanlurey removed their request for review April 2, 2025 15:29

-  /// Do not attempt to reboot device.
+  /// Do not automatically reboot the device after a certain number of tests.
+  /// In LUCI, every device is automatically restarted after a set number of tests.
+  /// Disabling the automatic reboot can be useful when running tests locally in a loop.

	'Do not attempt to reboot the device after every 30 test runs',
	'Do not automatically reboot the device after a certain number of tests',

Add flag to disable automatic device reboot for DeviceLab tests #164184

Are you sure you want to change the base?

Add flag to disable automatic device reboot for DeviceLab tests #164184

Conversation

harri35 commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

google-cla bot commented Feb 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harri35 commented Feb 26, 2025

Uh oh!

harri35 commented Feb 27, 2025

Uh oh!

harri35 commented Feb 27, 2025

Uh oh!

flutter-dashboard bot commented Mar 3, 2025

Uh oh!

Piinks commented Jun 4, 2025

Uh oh!

harri35 commented Jun 5, 2025

Uh oh!

Piinks commented Jun 11, 2025

Uh oh!

ergosarapu commented Jun 16, 2025

Uh oh!

Uh oh!

harri35 commented Feb 26, 2025 •

edited

Loading