Closed
Description
This is part of #103542, which is migrating gallery tests to build+test model.
We have validated the separation logic works as expected: we can build the apk first (save the apk in a specific location), and run the test separately based on the pre-build apk. But this works only when running the test based on the apk build from the same task.
For example:
Task flutter_gallery__transition_perf:
- build step:
dart bin/test_runner.dart test -t flutter_gallery__transition_perf --task-args build --task-args "application-binary-path=/tmp"buildsapkto/tmp/app-profile.apk. - test step:
dart bin/test_runner.dart test -t flutter_gallery__transition_perf --task-args test --task-args "application-binary-path=/tmp"succeeds based on the pre-build apk.
However, if we are running the test step of another task (say: flutter_gallery__transition_perf_e2e) based on the apk from above task flutter_gallery__transition_perf. It fails (this is based on the draft PR):
[flutter_gallery__transition_perf_e2e] [STDOUT] stdout: Running "flutter pub get" in flutter_gallery... 1,927ms
[flutter_gallery__transition_perf_e2e] [STDOUT] stdout: Installing ../../../../../../../../tmp/app-profile.apk... 13.9s
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:56135/DzJgkSacTP0=/
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: VMServiceFlutterDriver: Isolate found with number: 2751992616529479
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: VMServiceFlutterDriver: Isolate is paused at start.
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: VMServiceFlutterDriver: Attempting to resume isolate
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: VMServiceFlutterDriver: Connected to Flutter application.
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: Unhandled exception:
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: DriverError: Error in Flutter application: Uncaught extension error while executing request_data: Null check operator used on a null value
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: #0 main.<anonymous closure> (file:///Users/keyonghan/Documents/work/flutter/dev/integration_tests/flutter_gallery/test_driver/transitions_perf.dart:49)
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: #1 CommandHandlerFactory._requestData (package:flutter_driver/src/common/handler_factory.dart:224)
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: #2 CommandHandlerFactory.handleCommand (package:flutter_driver/src/common/handler_factory.dart:165)
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: #3 FlutterDriverExtension.handleCommand (package:flutter_driver/src/extension/extension.dart:437)
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: #4 FlutterDriverExtension.call (package:flutter_driver/src/extension/extension.dart:368)
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: #5 BindingBase.registerServiceExtension.<anonymous closure> (package:flutter/src/foundation/binding.dart:856)
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: <asynchronous suspension>
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr:
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr:
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: #0 VMServiceFlutterDriver.sendCommand (package:flutter_driver/src/driver/vmservice_driver.dart:325:7)
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: <asynchronous suspension>
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: #1 FlutterDriver.requestData (package:flutter_driver/src/driver/driver.dart:551:39)
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: <asynchronous suspension>
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: #2 integrationDriver (package:integration_test/integration_test_driver.dart:76:29)
[flutter_gallery__transition_perf_e2e] [STDOUT] stderr: <asynchronous suspension>
[flutter_gallery__transition_perf_e2e] [STDERR] Task failed: Executable "/Users/keyonghan/Documents/work/flutter/bin/flutter" failed with exit code 1.
[flutter_gallery__transition_perf_e2e] [STDERR]
[flutter_gallery__transition_perf_e2e] [STDERR] Stack trace:
[flutter_gallery__transition_perf_e2e] [STDERR] package:flutter_devicelab/framework/utils.dart 94:3 fail
[flutter_gallery__transition_perf_e2e] [STDERR] package:flutter_devicelab/framework/utils.dart 363:5 _execute
[flutter_gallery__transition_perf_e2e] [STDERR] ===== asynchronous gap ===========================
[flutter_gallery__transition_perf_e2e] [STDERR] dart:async _CustomZone.registerBinaryCallback
[flutter_gallery__transition_perf_e2e] [STDERR] package:flutter_devicelab/framework/framework.dart 274:26 _TaskRunner._performTask.<fn>
[flutter_gallery__transition_perf_e2e] [STDERR] package:stack_trace Chain.capture
[flutter_gallery__transition_perf_e2e] [STDERR] package:flutter_devicelab/framework/framework.dart 273:11 _TaskRunner._performTask
[flutter_gallery__transition_perf_e2e] [STDERR] package:flutter_devicelab/framework/framework.dart 166:43 _TaskRunner.run
From the build args, I didn't find any difference between these two tasks, but I guess I am missing something. /cc @zanderso Do you have any insight about this issue? The expected behavior is all these gallery tasks could run test step successfully based on a pre-built APK from any of these tasks.