Workaround for Dart VM timeout #127875

vashworth · 2023-05-30T20:27:58Z

Workaround solution for: #121231
See #120808 (comment) Error Case 2 for more information.

Sometimes the ios-deploy process does not return the logs from the application. We've been unable to figure out why. This is a solution to workaround that by using idevicesyslog alongside ios-deploy as a backup in getting the log for the Dart VM url. As explained in #120808 (comment), when error case 2 happens, the idevicesyslog does successfully find the Dart VM.

Also, in the comments of the code it mentions syslog is not written on iOS 13+, this was added in response to this issue: #41133.

However, idevicesyslog does in fact work (at least for iOS 16), we use it to collect device logs for our CI tests already:

flutter/dev/devicelab/lib/framework/devices.dart

Lines 998 to 1006 in 1dc26f8

    
           Future<void> startLoggingToSink(IOSink sink, {bool clear = true}) async { 
        
             // Clear is not supported. 
        
             _loggingProcess = await startProcess( 
        
               idevicesyslogPath, 
        
               <String>['-u', deviceId, '--quiet'], 
        
               environment: <String, String>{ 
        
                 'DYLD_LIBRARY_PATH': dyldLibraryPath, 
        
               }, 
        
             );

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is test-exempt.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

vashworth · 2023-05-30T21:59:16Z

packages/flutter_tools/lib/src/ios/devices.dart

+  /// https://github.com/flutter/flutter/issues/121231
+  bool get useBothLogDeviceReaders {
+    final String? user = _platform.environment['USER'];
+    if (user != null && user == 'swarming' && _majorSdkVersion >= 16) {


I tried to target just our CI bots by checking if the user is swarming but I'm unsure if this is a good way to do it.

can you instead add a helper method to class FlutterCommand, that looks something like this:

void usesCiFlag() { argParser.addFlag( 'ci', defaultsTo: false, help: 'Enable a set of CI-specific test debug settings.', hide: !verboseHelp, ) }

And then call that from the relevant sub-commands. You'll need to also update whatever tests/runners invoke the flutter CLI tool, but this should let us keep these types of settings organized moving forward.

Somewhere around here: https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/runner/flutter_command.dart#L273

What do you think about it being a global option here?

flutter/packages/flutter_tools/lib/src/runner/flutter_command_runner.dart

Lines 24 to 128 in b479691

FlutterCommandRunner({ bool verboseHelp = false }) : super(

'flutter',

'Manage your Flutter app development.\n'

'\n'

'Common commands:\n'

'\n'

' flutter create <output directory>\n'

' Create a new Flutter project in the specified directory.\n'

'\n'

' flutter run [options]\n'

' Run your Flutter application on an attached device or in an emulator.',

) {

argParser.addFlag('verbose',

abbr: 'v',

negatable: false,

help: 'Noisy logging, including all shell commands executed.\n'

'If used with "--help", shows hidden options. '

'If used with "flutter doctor", shows additional diagnostic information. '

'(Use "-vv" to force verbose logging in those cases.)');

argParser.addFlag('prefixed-errors',

negatable: false,

help: 'Causes lines sent to stderr to be prefixed with "ERROR:".',

hide: !verboseHelp);

argParser.addFlag('quiet',

negatable: false,

hide: !verboseHelp,

help: 'Reduce the amount of output from some commands.');

argParser.addFlag('wrap',

hide: !verboseHelp,

help: 'Toggles output word wrapping, regardless of whether or not the output is a terminal.',

defaultsTo: true);

argParser.addOption('wrap-column',

hide: !verboseHelp,

help: 'Sets the output wrap column. If not set, uses the width of the terminal. No '

'wrapping occurs if not writing to a terminal. Use "--no-wrap" to turn off wrapping '

'when connected to a terminal.');

argParser.addOption('device-id',

abbr: 'd',

help: 'Target device id or name (prefixes allowed).');

argParser.addFlag('version',

negatable: false,

help: 'Reports the version of this tool.');

argParser.addFlag('machine',

negatable: false,

hide: !verboseHelp,

help: 'When used with the "--version" flag, outputs the information using JSON.');

argParser.addFlag('color',

hide: !verboseHelp,

help: 'Whether to use terminal colors (requires support for ANSI escape sequences).',

defaultsTo: true);

argParser.addFlag('version-check',

defaultsTo: true,

hide: !verboseHelp,

help: 'Allow Flutter to check for updates when this command runs.');

argParser.addFlag('suppress-analytics',

negatable: false,

help: 'Suppress analytics reporting for the current CLI invocation.');

argParser.addFlag('disable-telemetry',

negatable: false,

help: 'Disable telemetry reporting each time a flutter or dart '

'command runs, until it is re-enabled.');

argParser.addFlag('enable-telemetry',

negatable: false,

help: 'Enable telemetry reporting each time a flutter or dart '

'command runs.');

argParser.addOption('packages',

hide: !verboseHelp,

help: 'Path to your "package_config.json" file.');

if (verboseHelp) {

argParser.addSeparator('Local build selection options (not normally required):');

}

argParser.addOption('local-engine-src-path',

hide: !verboseHelp,

help: 'Path to your engine src directory, if you are building Flutter locally.\n'

'Defaults to \$$kFlutterEngineEnvironmentVariableName if set, otherwise defaults to '

'the path given in your pubspec.yaml dependency_overrides for $kFlutterEnginePackageName, '

'if any.');

argParser.addOption('local-engine',

hide: !verboseHelp,

help: 'Name of a build output within the engine out directory, if you are building Flutter locally.\n'

'Use this to select a specific version of the engine if you have built multiple engine targets.\n'

'This path is relative to "--local-engine-src-path" (see above).');

argParser.addOption('local-web-sdk',

hide: !verboseHelp,

help: 'Name of a build output within the engine out directory, if you are building Flutter locally.\n'

'Use this to select a specific version of the web sdk if you have built multiple engine targets.\n'

'This path is relative to "--local-engine-src-path" (see above).');

if (verboseHelp) {

argParser.addSeparator('Options for testing the "flutter" tool itself:');

}

argParser.addFlag('show-test-device',

negatable: false,

hide: !verboseHelp,

help: 'List the special "flutter-tester" device in device listings. '

'This headless device is used to test Flutter tooling.');

argParser.addFlag('show-web-server-device',

negatable: false,

hide: !verboseHelp,

help: 'List the special "web-server" device in device listings.',

);

}

Our testing suite tests basically every flutter command, so if we want to make it more usable in the future for other purposes, I think that might be the way to go. However, it then begs the question of how to actually get the value. That could be per sub-command to do for itself or it could be a global/context thing. What do you think? I believe we're trying to move away from globals?

following up from an offline discussion, adding the command directly to the FlutterCommandRunner SGTM. We should avoid adding global/context getters, but we can call boolArg('ci') from within the sub-commands that want to check it.

christopherfujino · 2023-05-30T22:06:48Z

However, idevicesyslog does in fact work (at least for iOS 16), we use it to collect device logs for our CI tests already

Oh, this is really interesting. I wonder if a) idevicesyslog was updated to support the new log system; b) ios was updated to support the method idevicesyslog was using; or c) something else?

christopherfujino · 2023-05-30T22:18:16Z

Sometimes the ios-deploy process does not return the logs from the application. We've been unable to figure out why. This is a solution to workaround that by using idevicesyslog alongside ios-deploy as a backup in getting the log for the Dart VM url. As explained in #120808 (comment), when error case 2 happens, the idevicesyslog does successfully find the Dart VM.

Is the solution here actually to default back to using idevicesyslog?

vashworth · 2023-05-30T22:26:57Z

Oh, this is really interesting. I wonder if a) idevicesyslog was updated to support the new log system; b) ios was updated to support the method idevicesyslog was using; or c) something else?

I'm unsure. I looked through idevicesyslog git history and didn't see anything stand out.

Is the solution here actually to default back to using idevicesyslog?

Oh that's a possibility. I'm not sure we should default to idevicesyslog for everyone since it has been an issue in the past, but we could default to it for CI

christopherfujino · 2023-05-30T22:34:19Z

Is the solution here actually to default back to using idevicesyslog?

Oh that's a possibility. I'm not sure we should default to idevicesyslog for everyone since it has been an issue in the past, but we could default to it for CI

Yeah, let's land this change as is (modulo the new flag) for now. I would be worried about flipping the default without understanding why idevicesyslog started working again :)

bartekpacia · 2023-05-31T08:06:39Z

If I may chip in, I'd suggest you consider replacing ios-deploy with go-ios in the future.

A few copy-pasted arguments:

ios-deploy is a fragile dependency:

basically a single Objective-C file containing 3000+ lines of code with no tests.
looks like it's pretty much in maintenance mode (this probably stems from the fact that no one knows what's happening inside of it)
it's somewhat flaky

Meanwhile go-ios:

actively maintained (by a cool guy)
written in quite clean Go
outputs all logs in JSON
has its own Discord server.

I tried out go-ios and it already does most (if not all - didn't dig in that far) of what is already possible ios-deploy.

vashworth · 2023-05-31T16:46:24Z

If I may chip in, I'd suggest you consider replacing ios-deploy with go-ios in the future.

Thanks for the recommendation. I'll look into it!

christopherfujino · 2023-05-31T23:50:13Z

I'm scheduling retries of the firebase test failures, as those were upstream infra issues

vashworth · 2023-06-01T18:05:34Z

packages/flutter_tools/lib/src/ios/devices.dart

+  /// source is `idevicesyslog`. This is done because `ios-deploy` and
+  /// `idevicesyslog` often have different prefixes on non-flutter messages
+  /// and are often not critical for CI tests.
+  bool _excludeLog(String message, IOSDeviceLogSource source) {


FYI: I added this change because when ios-deploy fails to return the Dart VM url, it usually doesn't return any logs from the app. So I added this check so we can get other important logs from idevicesyslog other than just the Dart VM url. This way tests that use the device logs to check things (such as microbenchmarks_ios) will also work with this workaround.

vashworth · 2023-06-01T18:44:03Z

@christopherfujino friendly ping to let you know it's ready for review again

packages/flutter_tools/test/general.shard/args_test.dart

packages/flutter_tools/lib/src/runner/flutter_command_runner.dart

packages/flutter_tools/lib/src/ios/devices.dart

christopherfujino · 2023-06-01T19:16:22Z

packages/flutter_tools/lib/src/ios/devices.dart

+enum IOSDeviceLogSource {
+  iosDeploy,
+  idevicesyslog,
+  unifiedLogging,


what's unifiedLogging? Maybe these should have dartdocs.

Oh, does that mean we're grabbing both iosDeploy AND idevicesyslog in case we need to fall back to the latter?

No, it's actually like a third way to get device logs lol. I haven't looked into it too deeply but I believe it gets the logs from the VM

flutter/packages/flutter_tools/lib/src/ios/devices.dart

Lines 828 to 870 in ff33555

@override

set connectedVMService(FlutterVmService? connectedVmService) {

if (connectedVmService != null) {

_listenToUnifiedLoggingEvents(connectedVmService);

}

_connectedVMService = connectedVmService;

}

static const int minimumUniversalLoggingSdkVersion = 13;

Future<void> _listenToUnifiedLoggingEvents(FlutterVmService connectedVmService) async {

if (_majorSdkVersion < minimumUniversalLoggingSdkVersion) {

return;

}

try {

// The VM service will not publish logging events unless the debug stream is being listened to.

// Listen to this stream as a side effect.

unawaited(connectedVmService.service.streamListen('Debug'));

await Future.wait(<Future<void>>[

connectedVmService.service.streamListen(vm_service.EventStreams.kStdout),

connectedVmService.service.streamListen(vm_service.EventStreams.kStderr),

]);

} on vm_service.RPCError {

// Do nothing, since the tool is already subscribed.

}

void logMessage(vm_service.Event event) {

if (_iosDeployDebugger != null && _iosDeployDebugger!.debuggerAttached) {

// Prefer the more complete logs from the attached debugger.

return;

}

final String message = processVmServiceMessage(event);

if (message.isNotEmpty) {

_addToLinesController(message);

}

}

_loggingSubscriptions.addAll(<StreamSubscription<void>>[

connectedVmService.service.onStdoutEvent.listen(logMessage),

connectedVmService.service.onStderrEvent.listen(logMessage),

]);

}

packages/flutter_tools/lib/src/runner/flutter_command_runner.dart

christopherfujino · 2023-06-01T19:20:43Z

packages/flutter_tools/lib/src/runner/flutter_command.dart

@@ -1634,16 +1637,29 @@ Run 'flutter -h' (or 'flutter <command> -h') for available flutter commands and
  ///
  /// If no flag named [name] was added to the [ArgParser], an [ArgumentError]
  /// will be thrown.
-  bool boolArg(String name) => argResults![name] as bool;
+  bool boolArg(String name, {bool global = false}) {
+    if (global) {


Do we need to explicitly pass if we're checking for a global arg, or would it be safe to first check if (globalResults![name] == null) and if so, fall back to argResults?

Okay so took me forever to figure out, but we can't do it like this because when you access a key via [], if it doesn't find it, it will throw an error.
https://github.com/dart-lang/args/blob/a9543c021f9409832b1668f9256f247585362389/lib/src/arg_results.dart#L64-L66

We could alternatively, loop through the options to check for it, though.

if (globalResults!.options.contains(name)) { return globalResults![name] as bool; }

ahh yeah, forgot about that. Your snippet looks good to me.

Oof. Okay so that snippet doesn't work either because options get filtered out from the visible options attribute if they default to null. And if it tries to access a global option key from the argResults (and not the globalResults), it will throw the previously linked error because it can't find the key.
https://github.com/dart-lang/args/blob/a9543c021f9409832b1668f9256f247585362389/lib/src/arg_results.dart#L85

christopherfujino · 2023-06-01T19:22:06Z

packages/flutter_tools/lib/src/runner/flutter_command.dart

-  List<String> stringsArg(String name) {
+  List<String> stringsArg(String name, {bool global = false}) {
+    if (global) {
+      return globalResults![name]! as List<String>? ?? <String>[];


I know you were copying the existing code, but the existing code looks wrong: if I'm reading this right, the left hand side of the expression can never null because of the second !, and thus we would never hit the fallback literal.

You're right, in fact, this value should never be null.
https://github.com/dart-lang/args/blob/a9543c021f9409832b1668f9256f247585362389/lib/src/option.dart#L149-L153

christopherfujino

LGTM! Thanks for the cleanups too!

Roll Flutter from 3db9504 to 0b74153 (108 revisions) flutter/flutter@3db9504...0b74153 2023-06-03 [email protected] Roll Flutter Engine from 54293721816a to c838a1b05924 (2 revisions) (flutter/flutter#128161) 2023-06-03 [email protected] Roll Flutter Engine from 8769e9ce3a90 to 54293721816a (8 revisions) (flutter/flutter#128158) 2023-06-03 [email protected] Prefix for dart:ui multiview change (flutter/flutter#128152) 2023-06-02 [email protected] Roll dds dependency to 2.8.3 for expression evaluation upcoming fixes. (flutter/flutter#128124) 2023-06-02 [email protected] Roll Flutter Engine from f3f6a020595d to 8769e9ce3a90 (3 revisions) (flutter/flutter#128149) 2023-06-02 [email protected] Roll Flutter Engine from b4250acbf3ca to f3f6a020595d (3 revisions) (flutter/flutter#128147) 2023-06-02 [email protected] Reland "Remove obsolete drawShadow bounds workaround (#127052)" (flutter/flutter#127231) 2023-06-02 [email protected] Roll Flutter Engine from c6e9383f2143 to b4250acbf3ca (8 revisions) (flutter/flutter#128142) 2023-06-02 [email protected] Pre-migration for dart:ui multi view changes (flutter/flutter#128092) 2023-06-02 [email protected] Remove LiveTestRenderView (flutter/flutter#127882) 2023-06-02 [email protected] Roll Flutter Engine from 02d6fbb68b69 to c6e9383f2143 (15 revisions) (flutter/flutter#128119) 2023-06-02 [email protected] Tiny remove unnecessary method in text_selection.dart (flutter/flutter#127480) 2023-06-02 [email protected] Workaround for Dart VM timeout (flutter/flutter#127875) 2023-06-02 [email protected] Make --flutter-repo analyze whole repo (flutter/flutter#127990) 2023-06-02 [email protected] Roll Packages from f0513ae to 75085ed (3 revisions) (flutter/flutter#128121) 2023-06-02 [email protected] [Refactor] fix quote style in template (flutter/flutter#127762) 2023-06-02 [email protected] Roll Flutter Engine from 3a453f2ccb77 to 02d6fbb68b69 (14 revisions) (flutter/flutter#128089) 2023-06-02 [email protected] Fix typos in labels (flutter/flutter#128093) 2023-06-02 [email protected] Revert "Fix issue where DevTools would not be immediately available when using --start-paused" (flutter/flutter#128117) 2023-06-02 [email protected] Sync Lints (flutter/flutter#127976) 2023-06-02 [email protected] Updated TabBar and ToggleButtons examples (flutter/flutter#128088) 2023-06-02 [email protected] Text should still be centered when search bar height is less than 48 (flutter/flutter#128068) 2023-06-02 [email protected] Roll pub packages (flutter/flutter#128053) 2023-06-01 [email protected] Add `FilterChip.elevated`, `ChoiceChip.elevated`, & `ActionChip.elevated` variants (flutter/flutter#128049) 2023-06-01 [email protected] Updated custom ListTile examples (flutter/flutter#128071) 2023-06-01 [email protected] Fix issue where DevTools would not be immediately available when using --start-paused (flutter/flutter#126698) 2023-06-01 [email protected] Add error message when sliver overlap absorber is missing (flutter/flutter#128075) 2023-06-01 [email protected] Updated Menu examples (flutter/flutter#128080) 2023-06-01 [email protected] Add retry flag to flutter_test (flutter/flutter#125851) 2023-06-01 [email protected] [flutter_tools] Use process matcher for multidex test (flutter/flutter#127996) 2023-06-01 [email protected] Updated InputDecoratorExamples for M3 (flutter/flutter#128065) 2023-06-01 [email protected] Roll Flutter Engine from 7c0a639efd0a to 3a453f2ccb77 (1 revision) (flutter/flutter#128064) 2023-06-01 [email protected] migrate the package_autoroller from querying by label to title (flutter/flutter#128066) 2023-06-01 [email protected] Revised Floating Action Button examples (flutter/flutter#128058) 2023-06-01 [email protected] Roll Flutter Engine from fdd39c421fb9 to 7c0a639efd0a (2 revisions) (flutter/flutter#128056) 2023-06-01 [email protected] Roll Flutter Engine from c8e1a8d6fb57 to fdd39c421fb9 (1 revision) (flutter/flutter#128050) 2023-06-01 [email protected] [labeler] Remove use of any (flutter/flutter#128011) 2023-06-01 [email protected] [framework] remove reference to closed issue. (flutter/flutter#128007) 2023-06-01 [email protected] Roll Packages from 95bb793 to f0513ae (3 revisions) (flutter/flutter#128045) 2023-06-01 [email protected] Add fallback font to IconData class (flutter/flutter#127269) 2023-06-01 [email protected] [web] Assert route names start with / if using PathUrlStrategy (flutter/flutter#127986) 2023-06-01 [email protected] [tool] In `flutter doctor -v`, warn when Android Studio version could not be detected. (flutter/flutter#126395) 2023-06-01 [email protected] Roll Flutter Engine from 8339c71a963c to c8e1a8d6fb57 (1 revision) (flutter/flutter#128035) 2023-06-01 [email protected] Roll Flutter Engine from bc81133892a6 to 8339c71a963c (1 revision) (flutter/flutter#128034) 2023-06-01 [email protected] Roll Flutter Engine from e235f47164fa to bc81133892a6 (1 revision) (flutter/flutter#128026) 2023-06-01 [email protected] Fixes in Cupertino translations (flutter/flutter#127872) ...

add workaround by checking for Dart VM log from idevicesyslog

d1d9186

flutter-dashboard bot added platform-ios iOS applications specifically tool Affects the "flutter" command-line tool. See also t: labels. labels May 30, 2023

github-actions bot removed the tool Affects the "flutter" command-line tool. See also t: labels. label May 30, 2023

vashworth changed the title ~~add workaround by checking for Dart VM log from idevicesyslog~~ Workaround for Dart VM timeout May 30, 2023

flutter-dashboard bot added the tool Affects the "flutter" command-line tool. See also t: labels. label May 30, 2023

vashworth mentioned this pull request May 30, 2023

Mac_ios hot_mode_dev_cycle_ios__benchmark is 2.00% flaky, Process exited with status = -1 (0xffffffff) lost connection #120808

Closed

vashworth marked this pull request as ready for review May 30, 2023 21:55

vashworth requested a review from christopherfujino May 30, 2023 21:55

vashworth commented May 30, 2023

View reviewed changes

vashworth mentioned this pull request May 30, 2023

Tree closed on FTL infra failures #127884

Closed

adding CI flag

61eb1cd

github-actions bot removed the tool Affects the "flutter" command-line tool. See also t: labels. label Jun 1, 2023

vashworth added 2 commits June 1, 2023 10:55

add ci flag to our tests, inline bool check

94d0988

exclude ci flag from packages commands

f4073ca

github-actions bot added team Infra upgrades, team productivity, code health, technical debt. See also team: labels. tool Affects the "flutter" command-line tool. See also t: labels. labels Jun 1, 2023

vashworth added 3 commits June 1, 2023 11:45

add pacakges alias check also subcommand global args check

238a34d

fix format issues

1c67e54

allow flutter logs from either device log reader

41981ed

vashworth commented Jun 1, 2023

View reviewed changes