Add `SemanticsFinder` API for searching the Semantics tree #123634

pdblasi-google · 2023-03-28T20:21:31Z

Background

The Finder API is based off of navigating the Element tree. For interacting with the Semantics tree, we've been using tester.semantics.find (previously tester.getSemantics) which is based on the Finder API and is implemented as follows:

Use the Finder to get a singular Element
On that Element, use findRenderObject to get the closest render object
- The findRenderObject method will return the element if it happens to also be a render object, or the nearest descendant of the element that is a RenderObject
Once we have the closest RenderObject, find the related SemanticsNode to that RenderObject
- If the current RenderObject does not have a node, or that node is part of a merged node, we navigate up the RenderObject tree until we find a standalone SemanticsNode

The important parts here to note are:

We navigate down the RenderObject tree to find a RenderObject from an Element
- This RenderObject is not guaranteed to have a SemanticsNode
We then navigate up the RenderObject tree to find a SemanticsNode

The problem

TLDR;

Translating between the Element, RenderObject, and Semantics trees isn't consistent enough to be able to use the existing Finder API (which is based on the Element tree) to find specific SemanticsNodes in the Semantics tree.

Deep dive

The current method of finding SemanticsNodes related to the Element tree is a heuristic that doesn't consistently return the SemanticsNode that one would expect. For example, take a look at the following test:

void main() {
  testWidgets('Scratch', (WidgetTester tester) async {
    await tester.pumpWidget(const MaterialApp(
      home: Material(child: TextField()),
    ));

    expect(
      tester.semantics.find(find.byType(TextField)),
      containsSemantics(isTextField: true),
    );
  });
}

This test is very straightforward. When we try to find the SemanticsNode for a TextField the isTextField flag should be true, but this test ends up failing with the current implementation of find.

What went wrong

Looking at the implementation of find we can see a potential issue there that might resolve the issue. The findRenderObject method just looks for the closest descendent RenderObject. It doesn't really care whether or not that RenderObject has any semantics attached to it.

Turns out this is exactly the issue for the TextField case. The RenderObject returned is a RenderMouseRegion with no semantics attached to it. Looking at the RenderObject tree shows us that a couple steps further down the tree we can find a semantically relevant node:

RenderSemanticsAnnotations#efb04

RenderSemanticsAnnotations#efb04
│ needs compositing
│ creator: Semantics ← AnimatedBuilder ← IgnorePointer ←
│   TextFieldTapRegion ← MouseRegion ← TextField ← DefaultTextStyle
│   ← AnimatedDefaultTextStyle ← _InkFeatures-[GlobalKey#95de3 ink
│   renderer] ← NotificationListener<LayoutChangedNotification> ←
│   PhysicalModel ← AnimatedPhysicalModel ← ⋯
│ parentData: <none> (can use size)
│ constraints: BoxConstraints(w=800.0, h=600.0)
│ semantics node: SemanticsNode#4
│ size: Size(800.0, 600.0)

Which relates to the SemanticsNode we would expect:

SemanticsNode#4

└─SemanticsNode#4
    Rect.fromLTRB(0.0, 0.0, 800.0, 600.0)
    actions: tap
    flags: isTextField
    textDirection: ltr
    currentValueLength: 0

So, this gives us a potential solution! Lets adapt find to not just look for the closest descendant RenderObject but the closest descendant that also has semantics. Here's our new test with a fixedFind method:

void main() {
  testWidgets('Scratch', (WidgetTester tester) async {
    await tester.pumpWidget(const MaterialApp(
      home: Material(child: TextField()),
    ));

    debugDumpSemanticsTree();
    debugPrint(tester.semantics.find(find.byType(TextField)).toString());

    expect(
      fixedFind(find.byType(TextField)),
      containsSemantics(isTextField: true),
    );
  });
}

Implementation of fixedFind

fixedFind(Finder finder) {
    final Iterable<Element> candidates = finder.evaluate();
    final Element element = candidates.single;

    // Find the closest render object that has semantics data.
    final List<RenderObject> nextQueue = <RenderObject>[];
    RenderObject? renderObject = element.findRenderObject();
    SemanticsNode? result = renderObject?.debugSemantics;
    while (renderObject != null && result == null) {
      renderObject.visitChildren(nextQueue.add);
      if (nextQueue.isEmpty) {
        renderObject = null;
      } else {
        renderObject = nextQueue.removeAt(0);
        result = renderObject.debugSemantics;
      }
    }

    if (result == null) {
      throw StateError('No semantics data found.');
    }

    // Get the merged node if the related node is part of a merger
    while(result!.isMergedIntoParent) {
      result = result.parent;
    }
    return result;
}

And tada! The test now passes and find returns the SemanticsNode that we would expect from a TextField!

Success?

We solved the issue where we got the ancestor of the expected SemanticsNode! Perfect! Now let's make sure it works everywhere. Let's start with this test:

void main() {
  testWidgets('Scratch', (WidgetTester tester) async {
    await tester.pumpWidget( MaterialApp(
      home: ListView(
        children: [1, 2, 3, 4]
          .map((i) => Text('Testing $i', semanticsLabel: 'Semantics $i'))
          .toList(),
        ),
    ));

    expect(
      fixedFind(find.text('Testing 1')),
      containsSemantics(label: 'Testing 1'),
    );
  });
}

Nothing particularly out of the ordinary here, we've got a list of items, we're finding one of those items by text and making sure it has the correct semantics label. Unfortunately, this test now fails with a StateError saying: "No semantics data found.". Looking at the implementation of fixedFind, we can see that this happens when we've walked down the RenderObject tree and found no RenderObject that has a related SemanticsNode.

If you dig really deep, the root cause for this specific case is that being in an indexed scrollable moves the semantics up the render tree from the Element that is found by the find.text finder. If the text is not in an indexed scrollable, the semantics are down the tree where the implementation of fixedFind would expect it.

That just adds to the inconsistency of the Element -> RenderObject -> Semantics translation though. In the end, there isn't a heuristic for this translation that we could use that we could guarantee we're getting the "right" SemanticsNode for a given Element.

The solution

Create an API that is parallel to the Element based Finder API, but specifically for searching the Semantics tree.

SemanticsFinder will be based around the Finder API
- Possibly by extracting a FinderBase<TEvaluation> interface, though this isn't strictly necessary
The finders defined by flutter_test will be exposed through a CommonSemanticsFinders class, and be exposed through a find.semantics property on the existing global find property
- This mirrors the addition of the semantics property to WidgetController in 107866: Add support for verifying SemanticsNode ordering in widget tests #113133 while not requiring a secondary import, and not polluting the global namespace with a new semantics property.
For an initial implementation, we'll need the following finders:
- ancestor
  - Takes two SemanticsFinders and evaluates to all ancestors of of that meet the expectations of matching, optionally inclusive with matchesRoot
- descendant
  - Takes two SemanticsFinders and evaluates to all descendants of of that meet the expectations of matching, optionally inclusive with matchesRoot
- byLabel
  - Evaluates to all SemanticsNodes in the tree that has a label matching the given Pattern.
- byValue
  - Evaluates to all SemanticsNodes in the tree that has a value matching the given Pattern.
- byHint
  - Evaluates to all SemanticsNodes in the tree that has a hint matching the given Pattern.
- byAction
  - Evaluates to all SemanticsNodes in the tree that has the given SemanticsAction.
- byAnyAction
  - Evaluates to all SemanticsNodes in the tree that has at least one of the given SemanticsActions.
- byFlag
  - Evaluates to all SemanticsNodes in the tree that has the given SemanticsFlag.
- byAnyFlag
  - Evaluates to all SemanticsNodes in the tree that has at least one of the given SemanticsFlags.

The text was updated successfully, but these errors were encountered:

pdblasi-google · 2023-03-28T20:27:10Z

cc: @chunhtai @Hangyujin @dnfield @goderbauer

I've talked about implementing a separate set of finders specifically for Semantics before, but have been trying to avoid it and work within the existing Finder API so far.

While working on #117410, @goderbauer and I reached a tipping point where working within the existing Finder API just became too inconsistent for different reasonable use cases.

Any feedback you have on the above plan would be welcome!

chunhtai · 2023-03-28T21:01:06Z

I think the SemanticsFinder that target semantics tree can be useful for some use case. However, how would you use the API to test the use case like - I want to make sure my X widget has correct semantics Node. The SemanticsFinder may accidentally grab other node by mistake.

On the other hand, is it possible to do the following logic? we look down the renderobject until we hit the leaf or multichildrenderingobject, if we find one with semantics node, we return that node. If not we start look up parent until we find one.

This is kinda assuming how renderobject form semantics node explicitly or merge to parent

pdblasi-google · 2023-03-29T18:10:11Z

@chunhtai

I don't know that there really is a valid "I want to make sure Widget X has SemanticsNode Y" use case (with a specific exception below). The widget/element tree creates the render tree which composes the semantics tree, but there's not really a consistent way to find which SemanticsNode is related to which Widget.

The current implementation of find doesn't work if the findRenderObject method doesn't return a RenderObject that has semantics as was the case in the TextField example above.

The fixedFind implementation from above doesn't work if semantics are merged upwards for any reason in potentially two ways:

If the widget is close to a leaf of the tree, no semantics are found
If the widget is far from a leaf, the unbounded descendant search runs the risk of returning the semantics of a child, not its own semantics

I thought of two other possibilities, one of which was your suggestion of "if we don't find anything going down, go back up". This would solve the specific case in the writeup, but still has the unbounded descendant search issue and introduces a new issue where it could return a parent's SemanticsNode when it should have failed with a "No semantics" error.

The last possibility I thought of was "closest semantics in either direction" where you'd search both descendants and ancestors and return the SemanticsNode that took the least steps from the related RenderObject. Personally, I think this is very fragile and likely to return a parent's SemanticsNode incorrectly.

Long story short, I think anything we do to try to relate a Widget directly to a SemanticsNode via the semantics tree is going to be a heuristic at best and have a lot of edge cases where it either just won't work or will need to be special cased.

The exception I mentioned above is the case where a widget directly makes its own SemanticsNode. I'd like to (separately from this ticket) have a function that:

Takes a regular Finder
Gets the Element from that finder
Ensures that the element's widget is a RenderObjectWidget
Get the RenderObject from the widget
Get the semantics from that RenderObject
Fails with a StateError if any of the above steps don't work

This is close to the current find method, but completely removes the tree navigation which is what causes the current ambiguity. The StateError would drive users to use this new SemanticsFinder API to find specific nodes from the tree, or to further modify their finder to get the exact Element that can explicitly relate to a SemanticsNode. This new method would eventually act as a replacement for the current find method in conjunction with the API proposed above.

chunhtai · 2023-03-29T18:46:17Z

@pdblasi-google Thanks for the reply.

SemanticsFinder API

I think the idea sounds good to me. It has some overlaps with SemanticsTester/SemanticsTester.nodesWith though, but it is not too big of an issue since it is package private. We can consider merge them or deprecate SemanticsTester.

Original find API

If we want, we can expose SemanticsFragment for test only. With this, we can tell

If rendering object is merging up
If rendering object contribute to semantics at all
If rendering object formed a semantics node
If any of the children would create semantics node

Just throwing out ideas, and I agree this is probably not something we should focus on this issue. We can continue discussion somewhere else.

pdblasi-google · 2023-03-29T19:26:29Z

@chunhtai

Personally, I'd like to deprecate and remove SemanticsTester, though I'd also be up for making it public if the work here and in other recent tickets doesn't cover the use cases that SemanticsTester does. The fact that we had to create a class to make testing easier internally is kind of a hint that semantics testing has been harder than it should be for a while.

Thanks for the ideas, I wasn't aware of the _SemanticsFragment stuff, which may be able to fix find. It also reminded me that I didn't actually write up an issue for the find method bug itself... If you'd like, we can keep chatting on that topic over on #123719.

We may be able to replace find with something like a byWidget finder in this SemanticsFinder API when the time comes to help with consistency as well. That way any methods that make use of SemanticsFinder could make use of a fixed widget -> semantics translation.

goderbauer · 2023-03-29T23:04:26Z

byKey
Evaluates to all SemanticsNodes in the tree that have the specified key.

This finder seems odd? SemanticNodes don't have keys. Would these somehow use widget keys?

I want to make sure my X widget has correct semantics Node

I would argue that is the wrong way to look at it. Your semantics tests should make sure that the correct information are exposed to the semantics tree (e.g. that there is a SemanticsNode marked button with label X that when tapped does something) - not necessarily what widget contributed those semantics.

If we want, we can expose SemanticsFragment for test only.

I am hoping we can avoid that. Fragments are very complex, and it isn't a very user friendly API. The hope is that we can make testing for semantics easier with this.

Personally, I'd like to deprecate and remove SemanticsTester

Luckily, that's not a public API and just something used internally in framework tests. But I agree, it would be good if we wouldn't need it anymore and whatever we come up with here can replace it.

pdblasi-google · 2023-03-30T20:33:02Z

@goderbauer

byKey
Evaluates to all SemanticsNodes in the tree that have the specified key.

This finder seems odd? SemanticNodes don't have keys. Would these somehow use widget keys?

The SemanticsNode in the framework does define key, but that key is not passed to the engine. That said, looking at usages of the key property, it doesn't look like it's something that's passed through to the user APIs for the most part. It seems to be mostly an internal tool for caching.

Probably not as useful as I thought it'd be when looking over the options with that additional context... Removed!

* Pulled `FinderBase` out of `Finder` * `FinderBase` can be used for any object, not just elements * Terminology was updated to be more "find" related * Re-implemented `Finder` using `FinderBase<Element>` * Backwards compatibility maintained with `_LegacyFinderMixin` * Introduced base classes for SemanticsNode finders * Introduced basic SemanticsNode finders through `find.semantics` * Updated some relevant matchers to make use of the more generic `FinderBase` Closes #123634 Closes #115874

github-actions · 2023-08-24T22:00:40Z

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

pdblasi-google added a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. c: proposal A detailed proposal for a change to Flutter P2 Important issues not at the top of the work list labels Mar 28, 2023

pdblasi-google self-assigned this Mar 28, 2023

pdblasi-google changed the title ~~Add SemanticsFinder API for interacting with the Semantics tree~~ Add SemanticsFinder API for searching the Semantics tree Mar 28, 2023

pdblasi-google mentioned this issue Mar 28, 2023

Resolves 112413: Created APIs for triggering Semantics actions in tests. #117410

Closed

8 tasks

pdblasi-google mentioned this issue Mar 29, 2023

WidgetTester.semantics.find doesn't return expected node in some cases #123719

Open

pdblasi-google mentioned this issue Apr 25, 2023

Improve Finder APIs #115874

Closed

This was referenced May 18, 2023

Cleanup post FinderBase API introduction #127136

Closed

Adds SemanticsNode Finders for searching the semantics tree #127137

Merged

flutter-triage-bot bot added team-framework Owned by Framework team triaged-framework Triaged by Framework team labels Jul 8, 2023

auto-submit bot closed this as completed in #127137 Aug 10, 2023

github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `SemanticsFinder` API for searching the Semantics tree #123634

Add `SemanticsFinder` API for searching the Semantics tree #123634

pdblasi-google commented Mar 28, 2023 •

edited

Loading

pdblasi-google commented Mar 28, 2023

Uh oh!

chunhtai commented Mar 28, 2023

Uh oh!

pdblasi-google commented Mar 29, 2023 •

edited

Loading

Uh oh!

chunhtai commented Mar 29, 2023

Uh oh!

pdblasi-google commented Mar 29, 2023

Uh oh!

goderbauer commented Mar 29, 2023

Uh oh!

pdblasi-google commented Mar 30, 2023

Uh oh!

github-actions bot commented Aug 24, 2023

Uh oh!

Add SemanticsFinder API for searching the Semantics tree #123634

Add SemanticsFinder API for searching the Semantics tree #123634

Comments

pdblasi-google commented Mar 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

The problem

TLDR;

Deep dive

What went wrong

Success?

The solution

pdblasi-google commented Mar 28, 2023

Uh oh!

chunhtai commented Mar 28, 2023

Uh oh!

pdblasi-google commented Mar 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chunhtai commented Mar 29, 2023

Uh oh!

pdblasi-google commented Mar 29, 2023

Uh oh!

goderbauer commented Mar 29, 2023

Uh oh!

pdblasi-google commented Mar 30, 2023

Uh oh!

github-actions bot commented Aug 24, 2023

Uh oh!

Add `SemanticsFinder` API for searching the Semantics tree #123634

Add `SemanticsFinder` API for searching the Semantics tree #123634

pdblasi-google commented Mar 28, 2023 •

edited

Loading

pdblasi-google commented Mar 29, 2023 •

edited

Loading