Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix --local-engine for the new web/wasm mode #113759

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Oct 24, 2022
Merged

Fix --local-engine for the new web/wasm mode #113759

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
da9962b
Fix --local-engine for the new web wasm mode
mdebbar Oct 17, 2022
69410f6
fix libraries.json
mdebbar Oct 20, 2022
7867498
flip the order of lookup for dart sdk
mdebbar Oct 20, 2022
29a0042
Merge branch 'main' into flutter_tool_wasm
mdebbar Oct 20, 2022
2f307c9
minor change
mdebbar Oct 20, 2022
d8bdc9c
add tests
mdebbar Oct 24, 2022
5f102bb
remove prints
mdebbar Oct 24, 2022
d4b5d6a
fix analyzer error
mdebbar Oct 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 52 additions & 9 deletions packages/flutter_tools/lib/src/artifacts.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -768,24 +768,22 @@ class CachedLocalEngineArtifacts implements LocalEngineArtifacts {


@override
FileSystemEntity getHostArtifact(
HostArtifact artifact,
) {
FileSystemEntity getHostArtifact(HostArtifact artifact) {
switch (artifact) {
case HostArtifact.engineDartSdkPath:
final String path = _fileSystem.path.join(_hostEngineOutPath, 'dart-sdk');
final String path = _getDartSdkPath();
return _fileSystem.directory(path);
case HostArtifact.engineDartBinary:
final String path = _fileSystem.path.join(_hostEngineOutPath, 'dart-sdk', 'bin', _hostArtifactToFileName(artifact, _platform));
final String path = _fileSystem.path.join(_getDartSdkPath(), 'bin', _hostArtifactToFileName(artifact, _platform));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A use case that we should retain is using the flutter tool to drive a local engine build that has a modified Dart SDK. For example, imagine that a developer wants to use the flutter tool to drive an engine that has local modifications to dart2wasm.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should introduce a flag for this use case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hold on. Maybe we should reverse the lookup order? So we check for a dart sdk inside the out folder. If we can't find it, we look in the prebuilts folder. What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I might have been unclear. I think instead of this PR, the engine build should populate the directory that the tool is currently looking in.

Copy link
Contributor Author

@mdebbar mdebbar Oct 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output of the new --web build is platform-independent. We can build/zip it once then use the same on linux, mac and windows machines (in CI, for example).

Given that prebuilt dart sdks are platform-dependent, I'm not sure how we could do this while preserving the platform-independency of the build.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output of the new --web build is platform-independent. We can build/zip it once then use the same on linux, mac and windows machines (in CI, for example).

Yeah, that makes sense. I guess the way it'd have to work is that if you say --no-prebuilt-dart-sdk for the --web build, you'd build the Dart SDK with the default host toolchain.

Maybe the way forward here is to file an issue for that, and for now reverse the lookup order as you suggest. I'd also like to know if @jonahwilliams sees any issues with that approach.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! I created an issue: #113772 and reversed the lookup order.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly believe we should be rethinking this idea in general, at least for the web build if not more broadly. The --no-prebuilt-dart-sdk flag really complicates our build process and I think we should in general be moving away from it. Supporting both paths makes things very complicated.

In the web case, it is weird to put native host artifacts such as the dart sdk inside of the wasm output directory. There are ways of invoking the host toolchain from the wasm build if we want, but I question whether this is really worth the effort to support this mode. More broadly, it feels wrong for the dart sdk to be an actual output of the flutter engine build, even in the native case.

Editing code in third_party/dart inside the engine repo doesn't really feel like a great workflow to me personally, I'm not sure how many people rely on this. If someone is making changes to the dart sdk, I expect that they should have the dart sdk repo itself already checked out and are able to build using the normal build process for the dart sdk. If anything we should probably add support for using an arbitrary built dart sdk instead of coupling that to the engine build.

I'm not sure what the best long-term plan here is, but I think the main reason we expect the dart sdk to be in the engine build directory is for historical reasons before we were mostly using the sdk prebuilts. This might be a good opportunity to think about how these dependencies work and seeing if we can improve this process.

return _fileSystem.file(path);
case HostArtifact.dart2jsSnapshot:
final String path = _fileSystem.path.join(_hostEngineOutPath, 'dart-sdk', 'bin', 'snapshots', _hostArtifactToFileName(artifact, _platform));
final String path = _fileSystem.path.join(_getDartSdkPath(), 'bin', 'snapshots', _hostArtifactToFileName(artifact, _platform));
return _fileSystem.file(path);
case HostArtifact.dartdevcSnapshot:
final String path = _fileSystem.path.join(_dartSdkPath(_cache), 'bin', 'snapshots', _hostArtifactToFileName(artifact, _platform));
final String path = _fileSystem.path.join(_getDartSdkPath(), 'bin', 'snapshots', _hostArtifactToFileName(artifact, _platform));
return _fileSystem.file(path);
case HostArtifact.kernelWorkerSnapshot:
final String path = _fileSystem.path.join(_hostEngineOutPath, 'dart-sdk', 'bin', 'snapshots', _hostArtifactToFileName(artifact, _platform));
final String path = _fileSystem.path.join(_getDartSdkPath(), 'bin', 'snapshots', _hostArtifactToFileName(artifact, _platform));
return _fileSystem.file(path);
case HostArtifact.flutterWebSdk:
final String path = _getFlutterWebSdkPath();
Expand Down Expand Up @@ -908,7 +906,7 @@ class CachedLocalEngineArtifacts implements LocalEngineArtifacts {
return _fileSystem.path.join(_hostEngineOutPath, artifactFileName);
case Artifact.frontendServerSnapshotForEngineDartSdk:
return _fileSystem.path.join(
_hostEngineOutPath, 'dart-sdk', 'bin', 'snapshots', artifactFileName,
_getDartSdkPath(), 'bin', 'snapshots', artifactFileName,
);
}
}
Expand All @@ -923,6 +921,51 @@ class CachedLocalEngineArtifacts implements LocalEngineArtifacts {
buildMode == BuildMode.release ? 'flutter_patched_sdk_product' : 'flutter_patched_sdk');
}

String _getDartSdkPath() {
final String builtPath = _fileSystem.path.join(_hostEngineOutPath, 'dart-sdk');
if (_fileSystem.isDirectorySync(_fileSystem.path.join(builtPath, 'bin'))) {
return builtPath;
}

// If we couldn't find a built dart sdk, let's look for a prebuilt one.
final String prebuiltPath = _fileSystem.path.join(_getFlutterPrebuiltsPath(), _getPrebuiltTarget(), 'dart-sdk');
if (_fileSystem.isDirectorySync(prebuiltPath)) {
return prebuiltPath;
}

throw ToolExit('Unable to find a built dart sdk at: "$builtPath" or a prebuilt dart sdk at: "$prebuiltPath"');
}

String _getFlutterPrebuiltsPath() {
final String engineSrcPath = _fileSystem.path.dirname(_fileSystem.path.dirname(_hostEngineOutPath));
return _fileSystem.path.join(engineSrcPath, 'flutter', 'prebuilts');
}

String _getPrebuiltTarget() {
final TargetPlatform hostPlatform = _currentHostPlatform(_platform, _operatingSystemUtils);
switch (hostPlatform) {
case TargetPlatform.darwin:
return 'macos-x64';
case TargetPlatform.linux_arm64:
return 'linux-arm64';
case TargetPlatform.linux_x64:
return 'linux-x64';
case TargetPlatform.windows_x64:
return 'windows-x64';
case TargetPlatform.ios:
case TargetPlatform.android:
case TargetPlatform.android_arm:
case TargetPlatform.android_arm64:
case TargetPlatform.android_x64:
case TargetPlatform.android_x86:
case TargetPlatform.fuchsia_arm64:
case TargetPlatform.fuchsia_x64:
case TargetPlatform.web_javascript:
case TargetPlatform.tester:
throwToolExit('Unsupported host platform: $hostPlatform');
}
}

String _getFlutterWebSdkPath() {
return _fileSystem.path.join(engineOutPath, 'flutter_web_sdk');
}
Expand Down
41 changes: 24 additions & 17 deletions packages/flutter_tools/lib/src/asset.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,7 @@ class ManifestAssetBundle implements AssetBundle {
flutterManifest,
wildcardDirectories,
assetBasePath,
targetPlatform,
);

if (assetVariants == null) {
Expand Down Expand Up @@ -316,6 +317,7 @@ class ManifestAssetBundle implements AssetBundle {
// Do not track wildcard directories for dependencies.
<Uri>[],
packageBasePath,
targetPlatform,
packageName: package.name,
attributedPackage: package,
);
Expand Down Expand Up @@ -407,10 +409,11 @@ class ManifestAssetBundle implements AssetBundle {
final List<_Asset> materialAssets = <_Asset>[
if (flutterManifest.usesMaterialDesign)
..._getMaterialFonts(),
// Include the shaders unconditionally. They are small, and whether
// they're used is determined only by the app source code and not by
// the Flutter manifest.
..._getMaterialShaders(),
// For non-web platforms, include the shaders unconditionally. They are
// small, and whether they're used is determined only by the app source
// code and not by the Flutter manifest.
if (targetPlatform != TargetPlatform.web_javascript)
..._getMaterialShaders(),
];
for (final _Asset asset in materialAssets) {
final File assetFile = asset.lookupAssetFile(_fileSystem);
Expand Down Expand Up @@ -716,7 +719,8 @@ class ManifestAssetBundle implements AssetBundle {
PackageConfig packageConfig,
FlutterManifest flutterManifest,
List<Uri> wildcardDirectories,
String assetBase, {
String assetBase,
TargetPlatform? targetPlatform, {
String? packageName,
Package? attributedPackage,
}) {
Expand Down Expand Up @@ -750,18 +754,21 @@ class ManifestAssetBundle implements AssetBundle {
}
}

for (final Uri shaderUri in flutterManifest.shaders) {
_parseAssetFromFile(
packageConfig,
flutterManifest,
assetBase,
cache,
result,
shaderUri,
packageName: packageName,
attributedPackage: attributedPackage,
assetKind: AssetKind.shader,
);
// No shader compilation for the web.
if (targetPlatform != TargetPlatform.web_javascript) {
for (final Uri shaderUri in flutterManifest.shaders) {
_parseAssetFromFile(
packageConfig,
flutterManifest,
assetBase,
cache,
result,
shaderUri,
packageName: packageName,
attributedPackage: attributedPackage,
assetKind: AssetKind.shader,
);
}
}

// Add assets referenced in the fonts section of the manifest.
Expand Down
5 changes: 5 additions & 0 deletions packages/flutter_tools/lib/src/runner/local_engine.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,11 @@ class LocalEngineLocator {
// Determine the host engine directory associated with the local engine:
// Strip '_sim_' since there are no host simulator builds.
String _getHostEngineBasename(String localEngineBasename) {
if (localEngineBasename.startsWith('web_') || localEngineBasename.startsWith('wasm_')) {
// Don't modify the web local engine's basename.
return localEngineBasename;
}

String tmpBasename = localEngineBasename.replaceFirst('_sim_', '_');
tmpBasename = tmpBasename.substring(tmpBasename.indexOf('_') + 1);
// Strip suffix for various archs.
Expand Down
150 changes: 148 additions & 2 deletions packages/flutter_tools/test/general.shard/artifacts_test.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,13 @@ void main() {
.childDirectory('ios-arm64_armv7')
.childDirectory('Flutter.framework')
.createSync(recursive: true);
fileSystem
.directory('out')
.childDirectory('host_debug_unopt')
.childDirectory('dart-sdk')
.childDirectory('bin')
.createSync(recursive: true);

expect(
artifacts.getArtifactPath(
Artifact.flutterFramework,
Expand Down Expand Up @@ -325,6 +332,75 @@ void main() {
);
});

testWithoutContext('falls back to prebuilt dart sdk', () {
final String failureMessage = 'Unable to find a built dart sdk at:'
' "${fileSystem.path.join('/out', 'host_debug_unopt', 'dart-sdk')}"'
' or a prebuilt dart sdk at:'
' "${fileSystem.path.join('/flutter', 'prebuilts', 'linux-x64', 'dart-sdk')}"';

expect(
() => artifacts.getArtifactPath(Artifact.frontendServerSnapshotForEngineDartSdk),
throwsToolExit(message: failureMessage),
);
expect(
() => artifacts.getHostArtifact(HostArtifact.engineDartSdkPath),
throwsToolExit(message: failureMessage),
);
expect(
() => artifacts.getHostArtifact(HostArtifact.engineDartBinary),
throwsToolExit(message: failureMessage),
);
expect(
() => artifacts.getHostArtifact(HostArtifact.dart2jsSnapshot),
throwsToolExit(message: failureMessage),
);
expect(
() => artifacts.getHostArtifact(HostArtifact.dartdevcSnapshot),
throwsToolExit(message: failureMessage),
);
expect(
() => artifacts.getHostArtifact(HostArtifact.kernelWorkerSnapshot),
throwsToolExit(message: failureMessage),
);

fileSystem
.directory('flutter')
.childDirectory('prebuilts')
.childDirectory('linux-x64')
.childDirectory('dart-sdk')
.childDirectory('bin')
.createSync(recursive: true);

expect(
artifacts.getArtifactPath(Artifact.frontendServerSnapshotForEngineDartSdk),
fileSystem.path.join('/flutter', 'prebuilts', 'linux-x64', 'dart-sdk', 'bin',
'snapshots', 'frontend_server.dart.snapshot'),
);
expect(
artifacts.getHostArtifact(HostArtifact.engineDartSdkPath).path,
fileSystem.path.join('/flutter', 'prebuilts', 'linux-x64', 'dart-sdk'),
);
expect(
artifacts.getHostArtifact(HostArtifact.engineDartBinary).path,
fileSystem.path.join('/flutter', 'prebuilts', 'linux-x64', 'dart-sdk', 'bin', 'dart'),
);
expect(
artifacts.getHostArtifact(HostArtifact.dart2jsSnapshot).path,
fileSystem.path.join('/flutter', 'prebuilts', 'linux-x64', 'dart-sdk',
'bin', 'snapshots', 'dart2js.dart.snapshot'),
);
expect(
artifacts.getHostArtifact(HostArtifact.dartdevcSnapshot).path,
fileSystem.path.join('/flutter', 'prebuilts', 'linux-x64', 'dart-sdk',
'bin', 'snapshots', 'dartdevc.dart.snapshot'),
);
expect(
artifacts.getHostArtifact(HostArtifact.kernelWorkerSnapshot).path,
fileSystem.path.join('/flutter', 'prebuilts', 'linux-x64', 'dart-sdk',
'bin', 'snapshots', 'kernel_worker.dart.snapshot'),
);
});

testWithoutContext('getEngineType', () {
expect(
artifacts.getEngineType(TargetPlatform.android_arm, BuildMode.debug),
Expand All @@ -351,11 +427,81 @@ void main() {
operatingSystemUtils: FakeOperatingSystemUtils(),
);

expect(artifacts.getHostArtifact(HostArtifact.engineDartBinary).path, contains('.exe'));
fileSystem
.directory('out')
.childDirectory('host_debug_unopt')
.childDirectory('dart-sdk')
.childDirectory('bin')
.createSync(recursive: true);

expect(
artifacts.getHostArtifact(HostArtifact.engineDartBinary).path,
fileSystem.path.join('/out', 'host_debug_unopt', 'dart-sdk', 'bin', 'dart.exe'),
);
});

testWithoutContext('Looks up dart on linux platforms', () async {
expect(artifacts.getHostArtifact(HostArtifact.engineDartBinary).path, isNot(contains('.exe')));
fileSystem
.directory('/out')
.childDirectory('host_debug_unopt')
.childDirectory('dart-sdk')
.childDirectory('bin')
.createSync(recursive: true);

expect(
artifacts.getHostArtifact(HostArtifact.engineDartBinary).path,
fileSystem.path.join('/out', 'host_debug_unopt', 'dart-sdk', 'bin', 'dart'),
);
});

testWithoutContext('Finds dart-sdk in windows prebuilts', () async {
artifacts = LocalEngineArtifacts(
fileSystem.path.join(fileSystem.currentDirectory.path, 'out', 'android_debug_unopt'),
fileSystem.path.join(fileSystem.currentDirectory.path, 'out', 'host_debug_unopt'),
cache: cache,
fileSystem: fileSystem,
platform: FakePlatform(operatingSystem: 'windows'),
processManager: FakeProcessManager.any(),
operatingSystemUtils: FakeOperatingSystemUtils(),
);

fileSystem
.directory('/flutter')
.childDirectory('prebuilts')
.childDirectory('windows-x64')
.childDirectory('dart-sdk')
.childDirectory('bin')
.createSync(recursive: true);

expect(
artifacts.getHostArtifact(HostArtifact.engineDartBinary).path,
fileSystem.path.join('/flutter', 'prebuilts', 'windows-x64', 'dart-sdk', 'bin', 'dart.exe'),
);
});

testWithoutContext('Finds dart-sdk in macos prebuilts', () async {
artifacts = LocalEngineArtifacts(
fileSystem.path.join(fileSystem.currentDirectory.path, 'out', 'android_debug_unopt'),
fileSystem.path.join(fileSystem.currentDirectory.path, 'out', 'host_debug_unopt'),
cache: cache,
fileSystem: fileSystem,
platform: FakePlatform(operatingSystem: 'macos'),
processManager: FakeProcessManager.any(),
operatingSystemUtils: FakeOperatingSystemUtils(),
);

fileSystem
.directory('/flutter')
.childDirectory('prebuilts')
.childDirectory('macos-x64')
.childDirectory('dart-sdk')
.childDirectory('bin')
.createSync(recursive: true);

expect(
artifacts.getHostArtifact(HostArtifact.engineDartBinary).path,
fileSystem.path.join('/flutter', 'prebuilts', 'macos-x64', 'dart-sdk', 'bin', 'dart'),
);
});
});
}
Loading