Fix screenshot testing for flutter web integration_test #117114

kenzieschmoll · 2022-12-14T23:12:46Z

A complete example of screenshot testing still needs to be provided. This is better to be done once the screenshot testing workflow for integration tests is better thought out for web. Issue tracked at #86120.

kenzieschmoll · 2022-12-15T21:35:08Z

@yjbanov @Piinks This is ready for review. Please let me know what needs to be changed, and Kate, Yegor mentioned you might be able to help me move this PR over to using Skia Gold instead of the manual .png storage I have done here. Eager to get the changes to package:integration_test landed so we can use them in DevTools 👍 Thanks!

Piinks · 2022-12-15T23:57:42Z

Yegor mentioned you might be able to help me move this PR over to using Skia Gold

Yeah, we can't check image files into this repo. Unfortunately, I can't set up integration tests to send images to Gold right now.

Someone did wire them up for tests in exmaples/api though, you may be able to use #99298 as a guide. :)

kenzieschmoll · 2022-12-16T17:03:18Z

@Piinks do you know of any usages of flutter_goldens in integration tests?

kenzieschmoll · 2022-12-16T17:33:09Z

I followed the pattern in #99298 and added this to the test:

    await expectLater(
      find.byType(MaterialApp),
      matchesGoldenFile('platform_name.png'),
    );

When running the test:
flutter drive --driver=test_driver/extended_integration_test.dart --target=integration_test/extended_test.dart -d chrome

this is what was output:
Golden comparison requested for "platform_name.png"; skipping...

kenzieschmoll · 2022-12-16T17:37:15Z

packages/integration_test/example/test_driver/extended_integration_test.dart

+      List<int> screenshotBytes, [
+      Map<String, Object?>? args,
+    ]) async {
+      // Here is an example of what updating a golden image might look like.


@yjbanov @Piinks would it be acceptable to land this PR with the provided example in comments so that .png's are never added to this example? This still give some guidance or inspiration into how someone could hook up golden image testing if they weren't able to use Skia Gold.

Optionally, I can also remove this example and just leave the changes in this PR that are outside the example/ directory. We will do something like this for DevTools, since it looks like Skia Gold has not been hooked up yet for web integration tests (#86120). I'd like to unblock screenshot testing in integration tests for DevTools (even without skia gold support), so let me know what you think. Thanks.

Piinks · 2022-12-16T18:06:36Z

I followed the pattern in #99298 and added this to the test:
    await expectLater(
      find.byType(MaterialApp),
      matchesGoldenFile('platform_name.png'),
    );
When running the test: flutter drive --driver=test_driver/extended_integration_test.dart --target=integration_test/extended_test.dart -d chrome

this is what was output: Golden comparison requested for "platform_name.png"; skipping...

Did you push those changes? It won't work locally, it needs to be authenticated in CI.

kenzieschmoll · 2022-12-16T21:48:50Z

Did you push those changes? It won't work locally, it needs to be authenticated in CI.

Ah okay. Pushed them up now, so we'll see if it worked 👍

yjbanov · 2022-12-19T18:02:35Z

I'm not sure if flutter_goldens will work on the web. flutter test sets up a local server that knows how to take screenshots and compare them. I'm not aware of flutter drive or package:integration_test attempting to do that. package:integration_test wants us to perform the comparison in the onScreenshot callback, which is not part of the test code. We may need to look for something we can use in the callback. My first intuition was to use SkiaClient directly.

Alternatively, we can keep it simple for now and instead of integrating with Skia Gold, use package:image to decode the PNG and spot check a few pixels.

kenzieschmoll · 2022-12-19T18:22:05Z

Alternatively, we can keep it simple for now and instead of integrating with Skia Gold, use package:image to decode the PNG and spot check a few pixels.

Isn't the problem that we can't check PNGs into the codebase? how would we be able to use package:image to compare images without checking goldens in?

yjbanov · 2022-12-19T18:25:11Z

Isn't the problem that we can't check PNGs into the codebase? how would we be able to use package:image to compare images without checking goldens in?

Sorry, by "PNG" I meant the screenshot that the test takes and passes to the onScreenshot callback as screenshotBytes. The callback could parse the screenshot image and check a few known pixels in it.

kenzieschmoll · 2022-12-19T19:05:04Z

The callback could parse the screenshot image and check a few known pixels in it.

How would we get the "known" pixels? This sounds a bit hand wavy. Would we expect the average flutter developer to know how to parse out image "hot spots" from byte code and store those as constants to compare the new screenshotBytes to? Whatever is here in this example is likely the resource that users will look at to figure out how to add screenshot testing through package:integration_test to their app.

Could we possibly separate this PR into two tasks:

land the changes to packages/integration_test/lib (which un-breaks screenshot testing for flutter web and makes the onScreenshot handler more flexible)
Add an up to date example to the onScreenshot callback in examples/ on how to actually do the image comparison? It may make more sense to do this once a formal support for flutter_goldens on the web or some other recommended way is decided upon. For projects that are not Flutter, keeping .pngs in a repo and comparing the bytes may be a fine solution. Please see my question here: Fix screenshot testing for flutter web integration_test #117114 (comment).

I would like to shift this PR to address task 1 and leave task 2 for a later day when this user journey is better addressed.

kenzieschmoll · 2022-12-21T22:48:00Z

#117496 contains the same changes to dev/test.dart that this PR does (the changes to add the formerly failing tests in integration_test/example to Linux_web_long_running_tests on the CI). #117496 demonstrates that these tests are failing at master, but passing in this PR after the changes contained here.

packages/integration_test/lib/common.dart

yjbanov

…ter/flutter#117114)

* 95ff839 Roll Plugins from 54fc206 to 2dd85ec (3 revisions) (flutter/flutter#117535) * b8d5d9c Revert "Remove single-view assumption from widgets library (#117480)" (flutter/flutter#117545) * bd482eb fixes android_semantics_integration_test to expect long press for tooltip (flutter/flutter#117547) * 725c141 Fix screenshot testing for flutter web integration_test (flutter/flutter#117114) * 999356b Remove single-view assumption from ScrollPhysics (flutter/flutter#117503) * 6eb002a Reland "Remove single-view assumption from widgets library (#117480)" (flutter/flutter#117549) * 08209b7 Explain how to test onSubmitted in its docs (flutter/flutter#117550) * 20bc2ba Roll Flutter Engine from a90c45db3f13 to 234ab4c1e9c4 (1 revision) (flutter/flutter#117555) * abd5217 Bump ossf/scorecard-action from 2.1.1 to 2.1.2 (flutter/flutter#117554) * 6781576 Reland iOS 16 context menu (flutter/flutter#117234) * daa2ecf Roll Flutter Engine from 234ab4c1e9c4 to ca0c843bf75f (11 revisions) (flutter/flutter#117563) * 6441a7d Roll Flutter Engine from ca0c843bf75f to 12badb54598d (2 revisions) (flutter/flutter#117566) * a0cecbe 3b5fb86a5 delete unused lib/src/engine/canvaskit/viewport_metrics.dart (flutter/engine#38474) (flutter/flutter#117572) * fbfda23 Roll Flutter Engine from 3b5fb86a5982 to 6295d9198da1 (3 revisions) (flutter/flutter#117576) * 3eca29b f2a071692 Roll Skia from 3e39affa3e1d to 45466d04ca49 (1 revision) (flutter/engine#38480) (flutter/flutter#117579) * 05914e7 948d0bffa Roll Dart SDK from 47f192463696 to 442614a6c1bb (1 revision) (flutter/engine#38481) (flutter/flutter#117581) * a34dec2 00995b7b9 Roll Skia from 45466d04ca49 to e206aa0c44f0 (4 revisions) (flutter/engine#38482) (flutter/flutter#117585) * 4ef24cf 238f40bb9 Roll Fuchsia Mac SDK from W0GUdjHi4gI48optN... to 9w7QDlttR9f7Gu7U6... (flutter/engine#38483) (flutter/flutter#117587) * a45a2f3 ad042d863 Roll Skia from e206aa0c44f0 to a8b7ce3b6391 (1 revision) (flutter/engine#38484) (flutter/flutter#117589)

* Fix screenshot testing for flutter web integration_test * update packages * fix method signature and todo * Run tests on CI * fix type * remove silences * Add docs * fix comment * fix whitespace * review comments

* 95ff839 Roll Plugins from 54fc206 to 2dd85ec (3 revisions) (flutter/flutter#117535) * b8d5d9c Revert "Remove single-view assumption from widgets library (#117480)" (flutter/flutter#117545) * bd482eb fixes android_semantics_integration_test to expect long press for tooltip (flutter/flutter#117547) * 725c141 Fix screenshot testing for flutter web integration_test (flutter/flutter#117114) * 999356b Remove single-view assumption from ScrollPhysics (flutter/flutter#117503) * 6eb002a Reland "Remove single-view assumption from widgets library (#117480)" (flutter/flutter#117549) * 08209b7 Explain how to test onSubmitted in its docs (flutter/flutter#117550) * 20bc2ba Roll Flutter Engine from a90c45db3f13 to 234ab4c1e9c4 (1 revision) (flutter/flutter#117555) * abd5217 Bump ossf/scorecard-action from 2.1.1 to 2.1.2 (flutter/flutter#117554) * 6781576 Reland iOS 16 context menu (flutter/flutter#117234) * daa2ecf Roll Flutter Engine from 234ab4c1e9c4 to ca0c843bf75f (11 revisions) (flutter/flutter#117563) * 6441a7d Roll Flutter Engine from ca0c843bf75f to 12badb54598d (2 revisions) (flutter/flutter#117566) * a0cecbe 3b5fb86a5 delete unused lib/src/engine/canvaskit/viewport_metrics.dart (flutter/engine#38474) (flutter/flutter#117572) * fbfda23 Roll Flutter Engine from 3b5fb86a5982 to 6295d9198da1 (3 revisions) (flutter/flutter#117576) * 3eca29b f2a071692 Roll Skia from 3e39affa3e1d to 45466d04ca49 (1 revision) (flutter/engine#38480) (flutter/flutter#117579) * 05914e7 948d0bffa Roll Dart SDK from 47f192463696 to 442614a6c1bb (1 revision) (flutter/engine#38481) (flutter/flutter#117581) * a34dec2 00995b7b9 Roll Skia from 45466d04ca49 to e206aa0c44f0 (4 revisions) (flutter/engine#38482) (flutter/flutter#117585) * 4ef24cf 238f40bb9 Roll Fuchsia Mac SDK from W0GUdjHi4gI48optN... to 9w7QDlttR9f7Gu7U6... (flutter/engine#38483) (flutter/flutter#117587) * a45a2f3 ad042d863 Roll Skia from e206aa0c44f0 to a8b7ce3b6391 (1 revision) (flutter/engine#38484) (flutter/flutter#117589)

flutter-dashboard bot added the f: integration_test The flutter/packages/integration_test plugin label Dec 14, 2022

kenzieschmoll marked this pull request as ready for review December 15, 2022 21:32

kenzieschmoll requested review from Piinks and yjbanov December 15, 2022 21:35

kenzieschmoll mentioned this pull request Dec 15, 2022

Create integration testing harness flutter/devtools#4934

Closed

kenzieschmoll commented Dec 16, 2022

View reviewed changes

kenzieschmoll force-pushed the integration-test branch from 24d1349 to 79a39da Compare December 20, 2022 17:27

Fix screenshot testing for flutter web integration_test

2dc553c

kenzieschmoll force-pushed the integration-test branch from 30b2225 to 2dc553c Compare December 21, 2022 00:21

kenzieschmoll added 2 commits December 20, 2022 16:31

update packages

f32c665

fix method signature and todo

91915f5

flutter-dashboard bot added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Dec 21, 2022

kenzieschmoll mentioned this pull request Dec 21, 2022

Add integration_test/example tests to the CI #117496

Closed

kenzieschmoll added 3 commits December 21, 2022 14:12

Run tests on CI

3889ff8

fix type

7291c96

remove silences

7a441e6

kenzieschmoll added 3 commits December 21, 2022 16:28

Add docs

1d0ce87

fix comment

5e0c273

fix whitespace

7fddfca

yjbanov reviewed Dec 22, 2022

View reviewed changes

packages/integration_test/lib/common.dart Outdated Show resolved Hide resolved

packages/integration_test/lib/common.dart Outdated Show resolved Hide resolved

packages/integration_test/lib/common.dart Outdated Show resolved Hide resolved

review comments

405edd8

yjbanov approved these changes Dec 22, 2022

View reviewed changes

kenzieschmoll added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 22, 2022

auto-submit bot merged commit 725c141 into flutter:master Dec 22, 2022

kenzieschmoll deleted the integration-test branch December 22, 2022 22:00

This was referenced Dec 23, 2022

Roll Flutter from 9fb1ae839e0a to a45a2f311990 (19 revisions) flutter/packages#3001

Merged

Roll Flutter from 9fb1ae839e0a to a45a2f311990 (19 revisions) flutter/plugins#6879

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 23, 2022

725c141 Fix screenshot testing for flutter web integration_test (flut…

f04be6d

…ter/flutter#117114)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Dec 23, 2022

725c141 Fix screenshot testing for flutter web integration_test (flut…

253483b

…ter/flutter#117114)

Fix screenshot testing for flutter web integration_test #117114

Fix screenshot testing for flutter web integration_test #117114

Uh oh!

Conversation

kenzieschmoll commented Dec 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kenzieschmoll commented Dec 15, 2022

Uh oh!

Piinks commented Dec 15, 2022

Uh oh!

kenzieschmoll commented Dec 16, 2022

Uh oh!

kenzieschmoll commented Dec 16, 2022

Uh oh!

kenzieschmoll Dec 16, 2022

Choose a reason for hiding this comment

Uh oh!

kenzieschmoll Dec 16, 2022

Choose a reason for hiding this comment

Uh oh!

Piinks commented Dec 16, 2022

Uh oh!

kenzieschmoll commented Dec 16, 2022

Uh oh!

yjbanov commented Dec 19, 2022

Uh oh!

kenzieschmoll commented Dec 19, 2022

Uh oh!

yjbanov commented Dec 19, 2022

Uh oh!

kenzieschmoll commented Dec 19, 2022

Uh oh!

kenzieschmoll commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yjbanov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kenzieschmoll commented Dec 14, 2022 •

edited

Loading

kenzieschmoll commented Dec 21, 2022 •

edited

Loading