Allow marking a golden check as flaky #113396

yjbanov · 2022-10-13T16:42:55Z

(this is a non-breaking non-API-changing variant of #111595)

Support marking a golden check as flaky. A flaky check will still generate a golden and report it to the current GoldenFileComparator, but it will not fail the test.

Public API changes

In order to keep the concept of "flaky golden" private to the Flutter repo, the public golden test API was extended in the following non-breaking ways:

MatchesGoldenFile now takes an optional comparator argument. This allows the test to instruct the matcher to use a flaky comparator rather than always use the default one (the default API has no notion of "flaky golden").
- https://github.com/flutter/flutter/pull/113396/files#diff-503cfe6d62ac4cd45c0b021f709e278bc212e3e185027e50dbecae7a4bc3417d
Introduce flutter_web_test_config.dart, an optional sibling to flutter_test_config.dart. Because on the web the golden comparator is split into two parts, one that runs in the browser, and on that runs outside the browser, in order to keep the concept of flaky golden private to the Flutter repo, that logic goes into flutter_web_test_config.dart rather than being hard-coded into any of the public tool or package code.
- https://github.com/flutter/flutter/pull/113396/files#diff-331e9cbbfbec535700e20268c037352a820584d2efcd04a7e0099814cb752937

Private changes

The following changes do not affect the public API:

The Skia Gold client was updated with new API that submits the golden to Skia Gold with a virtually unlimited failure threshold. The effect is that the test doesn't fail and it does not generate a triage, but it still allows Skia Gold to track generated screenshots over time. When a fix for flakiness lands, it's easy to go Skia Gold UI to confirm it and remove the isFlaky argument from the test.
- https://github.com/flutter/flutter/pull/113396/files#diff-672b414f234faa47812f9af289e8a9d1072d8f0b225c757575a965a27c29bccf
package:flutter_goldens acquired a new function matchesFlutterGolden that's similar to matchesGoldenFile but supports flaky goldens.
- https://github.com/flutter/flutter/pull/113396/files#diff-57dd08efb172adaa80f3c0c5f03dd5a8ab9d6bc3025c0a3ed69ff0dd0ffd62feR25
package:flutter_goldens provides a command handler for web tests that implements the isFlaky option.
- https://github.com/flutter/flutter/pull/113396/files#diff-57dd08efb172adaa80f3c0c5f03dd5a8ab9d6bc3025c0a3ed69ff0dd0ffd62feR73
package:flutter_goldens now implements a testExecutable for web tests providing flaky functionality (previously the web version was empty).
- https://github.com/flutter/flutter/pull/113396/files#diff-8fa5d39f0c9a57621d61e4f9618715b392d509032919174103407f33cf5a1082R50

TODO

We have a check in the analyzer that ensures every matchesGoldenFile has a Tag at the top of the test file. It should probably be updated to ensure uses of matchesFlutterGolden are accounted for.
We'd probably want to migrate all uses of matchesGoldenFile in the framework tests to use this one for consistency, and have the analyzer enforce it.
Document the new flutter_web_test_config.dart file.
Add tests for isFlaky and new public API.

Fixes #111325

yjbanov · 2022-10-18T20:23:30Z

@Piinks the PR lacks documentation and tests. At this point I'm looking for an initial design review. Didn't want to spend time on minutiae without a high-level agreement first.

Piinks · 2022-10-20T22:09:45Z

packages/flutter_goldens/lib/src/flutter_goldens_io.dart

+  }
+}
+
+Future<GoldenFileComparator> _createComparator({ bool isFlaky = false }) async {


Why does the comparator need to know if something is flaky? Couldn't it just be a flag on compare? IIRC, we create a comparator for each test file, not each individual test.

That was my proposal in the previous PR, but that change turned out to be breaking and it made the isFlaky flag public, both aspects considered undesirable in review comments. However, I'm not against that approach. It will just require a small migration.

Sorry I meant the compare of the super class of Flutter's specific comparator (FlutterGoldenFileComparator).. not the public GoldenFileComparator.compare. Can we not add isFlaky to that comparison path?

Or a new FlutterGoldenFileComparator.compareFlaky method?

Oh, yeah, sub-classes can add extra parameters. I'll give it a shot.

This turned out not so simple: #113396 (comment)

Piinks · 2022-10-20T22:13:18Z

packages/flutter_goldens/lib/src/flutter_goldens_io.dart

+/// color. If the color is the same for all commits in the recent history, the
+/// golden is likely no longer flaky and [isFlaky] can be set back to false. If
+/// the color changes from commit to commit then it is still flaky.
+AsyncMatcher matchesFlutterGolden(Object key, { int? version, bool isFlaky = false }) {


We have a check in the analyzer that ensures every matchesGoldenFile has a Tag at the top of the test file. It should probably be updated to ensure uses of matchesFlutterGolden are accounted for.
Separately, we'd probably want to migrate all uses of matchesGoldenFile in the framework tests to use this one for consistency, and have the analyzer enforce it.

Also, the flutter/flutter wiki page on golden file testing for the framework will need to be updated.

Cool. I'm going to add it to my TODO list, but will wait until we converge on the overall design before starting the migration. Also, I imagine migrating all tests should probably done incrementally, since there are many usages of matchesGoldenFile.

Piinks · 2022-10-20T22:16:03Z

packages/flutter_goldens/lib/src/flutter_goldens_io.dart


 import 'package:file/file.dart';
 import 'package:file/local.dart';
 import 'package:flutter/foundation.dart';
 import 'package:flutter_goldens_client/skia_client.dart';
 import 'package:flutter_test/flutter_test.dart';
 import 'package:platform/platform.dart';
+import 'package:test_api/src/expect/async_matcher.dart' show AsyncMatcher; // ignore: implementation_imports


Would we be able to avoid this ignore if matchesGoldenFile was overridden instead?

matchesGoldenFile is a top-level function. I can't think of a way to override it. Any ideas?

You could use the same name and hide it from the namespace? Not sure. If there is a way to not expose this and have this ignore that would be ideal.

It's exposed from package:flutter_test/flutter_test.dart. According to git grep there are 874 imports of flutter_test.dart. Does the benefit of hiding it outweigh the cost of the noise across the codebase?

I dunno! Just brainstorming. I think in general there are a lot of options here. :)

Piinks · 2022-10-20T22:17:41Z

packages/flutter_goldens/lib/src/flutter_goldens_io.dart

+  final GoldenFileComparator nonFlakyComparator = await _createComparator();
+  _flakyGoldenFileComparator = await _createComparator(isFlaky: true);
+
+  // Assign this last because FlutterLocalFileComparator.fromDefaultComparator


I am not sure I follow, is this a bug?

I admit, I don't understand the golden design well enough to tell. The problematic line is

flutter/packages/flutter_goldens/lib/flutter_goldens.dart

Line 233 in 56c4e3c

defaultComparator ??= goldenFileComparator as LocalFileComparator;

So if I assign goldenFileComparator to anything other than LocalFileComparator the fromDefaultComparator fails the as cast, so I'm deferring the assignment till after all invocations of fromDefaultComparator.

I wonder if this could be a lot simpler if the actual comparator doesn't have anything to do with flakiness

Piinks · 2022-10-20T22:24:24Z

packages/flutter_tools/lib/src/test/flutter_web_goldens.dart

@@ -98,14 +98,14 @@ class TestGoldenComparator {
    return _processManager!.start(command, environment: environment);
  }

-  Future<String?> compareGoldens(Uri testUri, Uint8List bytes, Uri goldenKey, bool? updateGoldens) async {
+  Future<String?> compareGoldens(Uri testUri, Uint8List bytes, Uri goldenKey, bool? updateGoldens, Map<String, dynamic>? customProperties) async {


What is customProperties for?

For any custom properties needed by the custom comparator supplied by flutter_test_config.dart. In the case of Flutter's own goldens the only custom property is isFlaky. Without it, I'd have to hard-code the notion of isFlaky into the tool. Since we decided to limit this concept to Flutter's own goldens, I had to add a more generic way of passing custom properties. I'm open to other suggestions.

Piinks · 2022-10-20T22:46:24Z

packages/flutter_goldens/lib/src/flutter_goldens_io.dart

+          ? await _flakyGoldenFileComparator.compare(bytes, goldenKey)
+          : await goldenFileComparator.compare(bytes, goldenKey);


Yeah, like right here.. instead of having separate comparators, why not just add compareFlaky as a method to the base abstract FlutterGoldenFileComparator class from flutter_goldens? The post-, pre-submit, and local comparators will be able to override it if they need to do anything special in each test environment.

It looks like it could eliminate a bunch of code creating and managing multiple comparators.

I just realized that compare is also called by the matcher, which knows nothing about flakiness. So even if I have compareFlaky or an override of compare with isFlaky parameter, the matcher will not specify it 🤔

Aren't you creating a custom matcher though? I may not be understanding this properly.

I'm still reusing the existing matchers. I just teach them to optionally take a custom comparator and use it instead of the default one.

I could make a custom copy of the matcher too. But now I'm starting to realize that by replacing matchesGoldenFile, the comparator, and the matcher, I'd pretty much be duplicating all existing functionality. Is that OK? Or should we look for a way to reuse and customize existing functionality? An idea:

Go back to Allow marking a golden check as flaky #111595, but instead of hard-coding isFlaky, we expose an optional named String? tag argument in matchesGoldenFile.

This argument would be passed to GoldenFileComparator unchanged.

We document that this argument is not used by the default implementation of the golden API but may be used in conjunction with flutter_test_config.dart to customize the functionality of custom comparators.

In the Flutter repo tag: 'flaky' would be implemented by the custom comparator to implement flaky goldens, e.g.:

matchesGoldenFile('date_picker_test.time.initial.png', tag: 'flaky')

Properties of this approach:

Pro: it does not hard-code the notion of "flaky" into the public golden API.

Pro: it reuses all of the existing golden functionality, so that customizations like this are easy to implement.

Pro: it's flexible enough for developer to customize for different things (it's just a string; you can put whatever you want in it).

Con: String-based API is not very strongly typed (can be alleviated by checking the value at runtime).

Con: it's breaking, but easily fixable (simply pass customProperties to flutter_test_config.dart, or ignore it if it's not needed).

But now I'm starting to realize that by replacing matchesGoldenFile, the comparator, and the matcher, I'd pretty much be duplicating all existing functionality.

I don't think you need to replace all of these things. We already have custom comparators. They do not need to be copied and rewritten. If we create a matchesFlutterGolden method for the specific purpose of the flutter framework, then it can call on the custom comparator (of type FlutterGoldenFileComparator) that we already have to execute the test however we want it to. Because it is already encapsulated for our own specific uses. I feel like a lot of this is not necessary.

See #114026 as a rough example of what I mean.

Ah, you're right, I forgot that custom comparators can be reused, so we're not duplicating everything. Ok, moving in that direction.

Piinks · 2022-10-20T22:47:55Z

packages/flutter_goldens/lib/src/flutter_goldens_web.dart

+        'key': key,
+        'width': width.round(),
+        'height': height.round(),
+        'customProperties': <String, dynamic>{


Then you may not need these customProperties, if it is just a method rather than a unique comparator that handles the flaky case

Hmm, the way web goldens work is the comparator running in the browser isn't comparing anything. It just sends a command to the Flutter Tool and the tool forwards it to a comparator process. The comparator process needs to know whether to compare the golden as stable or as flaky. I'm currently using the customProperties to attach the isFlaky flag. I couldn't think of a different approach.

…/lib

Piinks · 2022-11-01T21:25:23Z

I forked this so I could play with it and get a better understanding of the web implementation. #114450 I made a few tweaks, let me know what you think. :)

yjbanov · 2022-11-08T00:36:47Z

Closing in favor of #114450

yjbanov force-pushed the flaky-golden-api-2 branch 3 times, most recently from 0add2ea to cbf2318 Compare October 17, 2022 23:47

yjbanov requested a review from Piinks October 18, 2022 20:21

Piinks reviewed Oct 20, 2022

View reviewed changes

yjbanov requested a review from Piinks October 25, 2022 00:47

allow marking a golden check as flaky

4cc45f9

yjbanov force-pushed the flaky-golden-api-2 branch from 6d75dee to 4cc45f9 Compare October 26, 2022 17:29

add matchesFlutterGolden to analyze.dart; no tags for flutter_goldens…

e7e1d7e

…/lib

yjbanov force-pushed the flaky-golden-api-2 branch from 3e8b71b to e7e1d7e Compare October 27, 2022 02:08

yjbanov closed this Nov 8, 2022

		? await _flakyGoldenFileComparator.compare(bytes, goldenKey)
		: await goldenFileComparator.compare(bytes, goldenKey);

Allow marking a golden check as flaky #113396

Allow marking a golden check as flaky #113396

Uh oh!

Conversation

yjbanov commented Oct 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Public API changes

Private changes

TODO

Uh oh!

yjbanov commented Oct 18, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Piinks commented Nov 1, 2022

Uh oh!

yjbanov commented Nov 8, 2022

Uh oh!

Uh oh!

yjbanov commented Oct 13, 2022 •

edited

Loading