[flutter_migrate] Compute #2734

GaryQian · 2022-10-24T20:05:00Z

Adds compute.dart which performs the bulk of the migrate tasks.

The migration is tracked as a MigrateContext which stores the MigrateResult as well as any intermediate state the migration needs.

The entry point is computeMigration(...) which begins the migration process. This is called by the start command which is the first command to be called by users.

As seen in the dart docs:

/// Computes the changes that migrates the current flutter project to the target revision.
///
/// This is the entry point to the core migration computations.
///
/// This method attempts to find a base revision, which is the revision of the Flutter SDK
/// the app was generated with or the last revision the app was migrated to. The base revision
/// typically comes from the .metadata, but for legacy apps, the config may not exist. In
/// this case, we fallback to using the revision in .metadata, and if that does not exist, we
/// use the target revision as the base revision. In the final fallback case, the migration should
/// still work, but will likely generate slightly less accurate merges.
///
/// Operations the computation performs:
///
///  - Parse .metadata file
///  - Collect revisions to use for each platform
///  - Download each flutter revision and call `flutter create` for each.
///  - Call `flutter create` with target revision (target is typically current flutter version)
///  - Diff base revision generated app with target revision generated app
///  - Compute all newly added files between base and target revisions
///  - Compute merge of all files that are modified by user and flutter
///  - Track temp dirs to be deleted
///
/// Structure: This method builds upon a MigrateResult instance

GaryQian · 2022-10-25T18:57:37Z

This PR depends on #2723 for environment.dart but is otherwise independent

christopherfujino · 2022-11-07T20:13:05Z

packages/flutter_migrate/lib/src/compute.dart

+  }
+  for (final String dir in _skippedDirectories) {
+    if (localPath.startsWith(
+        '${dir.replaceAll('/', fileSystem.path.separator)}${fileSystem.path.separator}')) {


isn't localPath absolute? so it would never start with, for example '.dart_tool', right? Or am I reading this wrong?

No, in this PR, localPath is relative to the project root.

Ahh, ok, I see you're right.

Similarly I recommend passing both _skippedDirectories and localPath through canonicalize() from package:path.

christopherfujino · 2022-11-07T21:36:27Z

This code doesn't even compile for me:

~/git/packages/packages/flutter_migrate$ dart test test/compute_test.dart
00:01 +0 -1: loading test/compute_test.dart [E]
  Failed to load "test/compute_test.dart":
  test/compute_test.dart:62:72: Error: Too few positional arguments: 2 required, 1 given.
          await FlutterToolsEnvironment.initializeFlutterToolsEnvironment(logger);
                                                                         ^
  lib/src/environment.dart:31:42: Context: Found this candidate, but the arguments don't match.
    static Future<FlutterToolsEnvironment> initializeFlutterToolsEnvironment(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

GaryQian · 2022-11-07T21:38:08Z

This is dependent on #2723, which just landed today. I should have just rebased it. Due to the review comments, I have changed the dependency API a bit, let me fix it real quick.

christopherfujino · 2022-11-07T21:39:54Z

This is dependent on #2723, which just landed today. I should have just rebased it. Due to the review comments, I have changed the dependency API a bit, let me fix it real quick.

Ohh yeah sorry, you mentioned that

packages/flutter_migrate/lib/src/compute.dart

stuartmorgan-g · 2022-11-07T21:23:00Z

packages/flutter_migrate/lib/src/compute.dart

+  'lib', // Files here are always user owned and we don't want to overwrite their apps.
+  'test', // Files here are typically user owned and flutter-side changes are not relevant.
+  'assets', // Common directory for user assets.
+  'build' // Build artifacts


It would help readability to make the comments here all use the same form (consistency about capitalization and punctuation, consistency about whether to say "ignore the" given that's all ignoring, consistency about whether it's describing the files or the reason for excluding them).

Also, I would suggest keeping this list alphabetized so it's easier to see what's there.

stuartmorgan-g · 2022-11-07T21:25:07Z

packages/flutter_migrate/lib/src/compute.dart

+];
+
+bool _skipped(String localPath, FileSystem fileSystem,
+    {Set<String>? blacklistPrefixes}) {


skippedPrefixes

packages/flutter_migrate/lib/src/compute.dart

stuartmorgan-g · 2022-11-07T21:41:27Z

packages/flutter_migrate/lib/src/compute.dart

+  status.pause();
+  logger.printStatus('Obtaining revisions.',
+      indent: 2, color: TerminalColor.grey);
+  status.resume();


Having UI and core logic intermixed is going to make this hard to maintain over time (and to extend to other uses, like a GUI tool). I would strongly recommend having a state system that this communicates to, and have that state drive UI from completely separate code.

stuartmorgan-g · 2022-11-07T21:42:23Z

packages/flutter_migrate/lib/src/compute.dart

+  }
+  status.stop();
+  return context.migrateResult;
+}


This function is quite large, and doing a lot of things; extracting smaller, well-named helpers for individual parts would help with understanding the overall flow.

stuartmorgan-g · 2022-11-07T21:42:53Z

packages/flutter_migrate/lib/src/compute.dart

+  return '5391447fae6209bb21a89e6a5a6583cac1af9b4b';
+}
+
+abstract class MigrateFlutterProject {


This could use docs.

stuartmorgan-g · 2022-11-07T21:43:32Z

packages/flutter_migrate/lib/src/compute.dart

+      }
+    }
+  }
+}


Here too, consider extracting small helpers to allow someone to easily see the overall logical structure of the flow.

stuartmorgan-g · 2022-11-07T21:44:03Z

packages/flutter_migrate/lib/src/compute.dart

+  }
+}
+
+class MigrateTargetFlutterProject extends MigrateFlutterProject {


This could use docs explaining its role.

christopherfujino · 2022-11-16T18:22:00Z

packages/flutter_migrate/lib/src/flutter_project_metadata.dart

@@ -226,10 +226,10 @@ ${migrateConfig.getOutputFileString()}''';
 /// used to add support for new platforms, so the base and create revision may not always be the same.
 class MigrateConfig {
  MigrateConfig(
-      {Map<SupportedPlatform, MigratePlatformConfig>? platformConfigs,
+      {Map<SupportedPlatform?, MigratePlatformConfig>? platformConfigs,


how is a null key used here?

christopherfujino · 2022-11-16T18:22:50Z

packages/flutter_migrate/lib/src/flutter_project_metadata.dart

@@ -252,7 +252,7 @@ class MigrateConfig {
  /// Parses the project for all supported platforms and populates the [MigrateConfig]
  /// to reflect the project.
  void populate({
-    List<SupportedPlatform>? platforms,
+    List<SupportedPlatform?>? platforms,


i guess same question here, is null a valid platform?

See discussion below, changed to use a new FlutterProjectComponent enum that includes root as a value.

stuartmorgan-g · 2022-11-18T19:50:41Z

packages/flutter_migrate/lib/src/base/project.dart

-    final List<SupportedPlatform> platforms = includeRoot
-        ? <SupportedPlatform>[SupportedPlatform.root]
-        : <SupportedPlatform>[];
+  List<SupportedPlatform?> getSupportedPlatforms() {


It doesn't look like this should need to return a nullable generic.

stuartmorgan-g · 2022-11-18T19:51:49Z

packages/flutter_migrate/lib/src/flutter_project_metadata.dart

      this.unmanagedFiles = kDefaultUnmanagedFiles})
      : platformConfigs =
-            platformConfigs ?? <SupportedPlatform, MigratePlatformConfig>{};


This should be writeable more simply as:

MigrateConfig({ this.platformConfigs = const <...>{}, });

stuartmorgan-g · 2022-11-18T19:54:41Z

packages/flutter_migrate/lib/src/flutter_project_metadata.dart

        in platformConfigs.entries) {
      platformsString +=
-          '\n    - platform: ${entry.key.toString().split('.').last}\n      create_revision: ${entry.value.createRevision == null ? 'null' : "${entry.value.createRevision}"}\n      base_revision: ${entry.value.baseRevision == null ? 'null' : "${entry.value.baseRevision}"}';
+          '\n    - platform: ${entry.key == null ? 'root' : entry.key.toString().split('.').last}\n      create_revision: ${entry.value.createRevision == null ? 'null' : "${entry.value.createRevision}"}\n      base_revision: ${entry.value.baseRevision == null ? 'null' : "${entry.value.baseRevision}"}';


This is still conflating "root" with platforms, just at a different level.

Stepping back a bit (since I'm not familiar with the high level flow), what is the thing that you are trying to represent here? Is it directories that make up a project, including both the root and the platform subdirectories?

The purpose of this code is to track the Flutter git revision each project directory was flutter create-ed with. For this purpose only, the root directory is equivalent to any other platform-specific directory. It is equivalent because, like a platform directory, the root directory contains a subset of files that are created by the flutter create command. Upon creation, we log the flutter revision associated with the directory. Here, SupportedPlatform is a reference to directories within a project. As far as the migrate tool is concerned, a SupportedPlatform is just a directory, and it does not care about what the directory represents.

This tracking is to handle the case where the initial flutter create is called with, say 1.0.0, and then the windows platform is added using flutter 3.0. We would track different revisions for root+ios+android and windows.

Within the migrate tool, we also treat the root directory in the same way as platform directories. Once a project goes through a successful migration, we also track last successfully migrated revision in the same way.

It sounds like there should be a different class/enum for this that captures the concept then; something like FlutterProjectSection or FlutterProjectComponent. That way the confusion that comes from trying to express something other than a platform with SupportedPlatform is eliminated.

I've added FlutterProjectComponent, and there are places in the code where it is appropriate to use SupportedPlatform and others where FlutterProjectComponent makes more sense. Having both does indeed seem like a more reasonable solution than conflating the two.

stuartmorgan-g · 2022-11-18T19:57:02Z

packages/flutter_migrate/lib/src/migrate_logger.dart

+  // We keep a spinner going and print periodic progress messages
+  // to assure the developer that the command is still working due to
+  // the long expected runtime.
+  Status? status;


Why is this nullable? I'm not clear from reading this code what the distinction is between "never started" (null) and "started, but then stopped (non-null).

I'll change this to initialize the spinner in the constructor

GaryQian · 2022-11-30T20:09:55Z

@christopherfujino @stuartmorgan Comments addressed. Ready for another look.

packages/flutter_migrate/lib/src/compute.dart

christopherfujino

LGTM

stuartmorgan-g

I left a few small comments, but otherwise looks good! I didn't do a full review, since you have one already, so I just skimmed (mostly around areas where I'd left comments previously) so my previous review won't block landing.

stuartmorgan-g · 2022-12-05T21:17:52Z

packages/flutter_migrate/lib/src/custom_merge.dart

-    final Map<SupportedPlatform, MigratePlatformConfig> platformConfigs =
-        <SupportedPlatform, MigratePlatformConfig>{};
-    for (final MapEntry<SupportedPlatform, MigratePlatformConfig> entry
+    final Map<FlutterProjectComponent, MigratePlatformConfig> platformConfigs =


This should probably be called projectComponents, matching the type change.

stuartmorgan-g · 2022-12-05T21:18:38Z

packages/flutter_migrate/lib/src/custom_merge.dart

-        <SupportedPlatform, MigratePlatformConfig>{};
-    for (final MapEntry<SupportedPlatform, MigratePlatformConfig> entry
+    final Map<FlutterProjectComponent, MigratePlatformConfig> platformConfigs =
+        <FlutterProjectComponent, MigratePlatformConfig>{};


Should the value be a Platform-related thing if the key is no longer platform-specific? Does that type need a name change or split as well?

Yeah, it does. But I think I'll leave a TODO here and do it in a separate PR to sync it here and in the flutter_tools repo. Id prefer to keep these concepts synced.

stuartmorgan-g · 2022-12-05T21:19:11Z

packages/flutter_migrate/lib/src/flutter_project_metadata.dart

@@ -9,6 +9,37 @@ import 'base/file_system.dart';
 import 'base/logger.dart';
 import 'base/project.dart';

+enum FlutterProjectComponent {


This should have a documentation comment.

stuartmorgan-g · 2022-12-05T21:21:41Z

packages/flutter_migrate/lib/src/flutter_project_metadata.dart

+}
+
+extension FlutterProjectComponentExtension on FlutterProjectComponent {
+  SupportedPlatform? toSupportedPlatform() {


Maybe remove this? It doesn't appear to be used anywhere, and it's a remnant of conflating components and platforms. Code using this seems like it would be suspect for the same reason the earlier design was problematic.

stuartmorgan-g · 2022-12-05T21:24:10Z

packages/flutter_migrate/lib/src/compute.dart

+// final String canonicalizedLocalPath = canonicalize(localPath);
+// if (canonicalizedSkippedFiles.contains(canonicalizedLocalPath)) {
+//   return true;
+// }


Remove? We generally don't want to check in commented-out code, especially not without a comment explaining it.

Oops, missed this in the cleanup.

stuartmorgan-g · 2022-12-05T21:30:48Z

packages/flutter_migrate/lib/src/compute.dart

+/// False for files that should not be merged. Typically, images and binary files.
+bool _mergable(String localPath) {
+  for (final String ext in _doNotMergeFileExtensions) {
+    if (localPath.endsWith(ext) && !_alwaysMigrateFiles.contains(localPath)) {


There's no need to re-check _alwaysMigrateFiles in the loop, since that expression doesn't use ext.

It looks like this whole function could simplify to:

return _alwaysMigrateFiles.contains(localPath) || ! _doNotMergeFileExtensions.any((String ext) => localPath.endsWith(ext));

stuartmorgan-g · 2022-12-05T21:32:03Z

packages/flutter_migrate/lib/src/compute.dart

+    skippedPrefixes.add(platformToSubdirectoryPrefix(platform));
+  }
+  for (final SupportedPlatform platform in platforms) {
+    if (platform != null) {


This can't happen any more; it's a non-nullable type now.

github-actions bot added the p: flutter_migrate label Oct 24, 2022

GaryQian force-pushed the migratecompute branch from 0f3eeba to b6dce1c Compare October 25, 2022 18:51

GaryQian mentioned this pull request Oct 28, 2022

[flutter_migrate] Start command and executables #2735

Merged

GaryQian requested review from stuartmorgan-g, zanderso and christopherfujino November 7, 2022 18:25

GaryQian marked this pull request as ready for review November 7, 2022 18:25

christopherfujino reviewed Nov 7, 2022

View reviewed changes

GaryQian force-pushed the migratecompute branch from b6dce1c to 6e0a57b Compare November 7, 2022 20:44

stuartmorgan-g requested changes Nov 7, 2022

View reviewed changes

GaryQian requested a review from stuartmorgan-g November 14, 2022 22:13

GaryQian force-pushed the migratecompute branch from 04c4245 to 600d98a Compare November 15, 2022 20:11

christopherfujino reviewed Nov 16, 2022

View reviewed changes

stuartmorgan-g reviewed Nov 18, 2022

View reviewed changes

GaryQian force-pushed the migratecompute branch from b4da944 to 1ecebdc Compare November 30, 2022 05:22

GaryQian requested review from stuartmorgan-g and christopherfujino November 30, 2022 07:06

christopherfujino reviewed Dec 1, 2022

View reviewed changes

packages/flutter_migrate/lib/src/compute.dart Outdated Show resolved Hide resolved

christopherfujino reviewed Dec 1, 2022

View reviewed changes

packages/flutter_migrate/lib/src/compute.dart Show resolved Hide resolved

christopherfujino reviewed Dec 1, 2022

View reviewed changes

packages/flutter_migrate/lib/src/compute.dart Outdated Show resolved Hide resolved

christopherfujino reviewed Dec 1, 2022

View reviewed changes

packages/flutter_migrate/lib/src/compute.dart Show resolved Hide resolved

GaryQian requested a review from christopherfujino December 2, 2022 18:59

christopherfujino approved these changes Dec 2, 2022

View reviewed changes

Compute

4b8830e

GaryQian added 21 commits December 5, 2022 13:12

Licenses for test

259f131

Merge with depenencies

fbc297d

Address comments p1

49707f7

Separate out logging from logic

4c240b4

Refactor into smaller methods

5fd5126

Improve logging and verbose usages

86a0ca9

Diferentiate merge skip vs total skip

995ed79

More docs

4863308

Remove additional skip

829f6d5

Fix custom merge tests

74b3890

Mocked environment

5a8d73f

Formatting

c58deec

TImeouts

db78339

Use separate enum to address project directories including root

0ed50ac

tests passing

79a7243

Fix analyzer

01119b1

address comments, formatting

09662e8

Test robustness

86d059b

Fix update locks test

44b1de3

formatting

9c20b09

logging for CI test failures

2c7bdfd

GaryQian force-pushed the migratecompute branch from a570d62 to 2c7bdfd Compare December 5, 2022 21:12

stuartmorgan-g approved these changes Dec 5, 2022

View reviewed changes

GaryQian added 5 commits December 5, 2022 14:05

Canonicalize test paths

8e0df1c

Canonizalize both sides of tests

451538c

Address comments, fix tests

b9a2cca

Formatting

a6cd77d

Gradle locks test

5016648

GaryQian added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 6, 2022

auto-submit bot merged commit 57e388d into flutter:main Dec 6, 2022

[flutter_migrate] Compute #2734

[flutter_migrate] Compute #2734

Uh oh!

Conversation

GaryQian commented Oct 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GaryQian commented Oct 25, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christopherfujino commented Nov 7, 2022

Uh oh!

GaryQian commented Nov 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

christopherfujino commented Nov 7, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GaryQian Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GaryQian commented Nov 30, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

christopherfujino left a comment

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

GaryQian commented Oct 24, 2022 •

edited

Loading

GaryQian commented Nov 7, 2022 •

edited

Loading

GaryQian Nov 28, 2022 •

edited

Loading