I think we want to avoid mandatory full semantics matching because taken to even a mild extreme, that becomes just a different way to encode a golden test.

The advantage of the isBeforeX and isAfterX matchers is that they intentionally limited and very, uh, for like of better term semantic. Though I could see cases where you'd want to verify the complete order - for example, the old time picker where each number was on a clock face. We had to check that it went 1 -> 12 instead of jumping around the clock.

It also depends on the use cases, what are some examples outside of the repo where users are checking ordering?

@jonahwilliams Agreed about the full semantics concern – I'm leaning towards an option that just verifies a11y labels or something similarly minimal.

Re: use cases, I think any page with multiple elements would benefit from an element ordering test. Say I'm building a contact info form. I could write:

expect(find.text('Name'), isOrderedBefore(find.text('Address 1')));
expect(find.text('Address 1'), isOrderedBefore(find.text('Address 2')));
expect(find.text('Address 2'), isOrderedBefore(find.text('City')));
expect(find.text('City'), isOrderedBefore(find.text('State')));
expect(find.text('State'), isOrderedBefore(find.text('ZIP')));

or, I could write

expect(
  find.byKey(ValueKey('myForm')),
  hasSemanticsElements([
    find.text('Name'),
    find.text('Address 1'),
    find.text('Address 2'),
    find.text('City'),
    find.text('State'),
    find.text('ZIP'),
  ]),
);

The second of those is a lot clearer and was much more enjoyable to write. I think this applies anywhere element ordering matters, which is everywhere IMO.

One problem folks frequently encounter, is that testing the order here sort of requires knowledge about how data is combined into semantic labels. For example, suppose they had two different widgets for the name field like Text('First') + Text('Name') for some reason...

And they wrote:

expect(
  find.byKey(ValueKey('myForm')),
  hasSemanticsElements([
    find.text('First'),
    find.text('Name'),
    find.text('Address 1'),
    find.text('Address 2'),
    find.text('City'),
    find.text('State'),
    find.text('ZIP'),
  ]),
);

In this case both the First and Name finder would correspond to the same semantics node. Does this pass? or fail? or just add a warning to the logs?

I guess this technically applies to isBefore and isAfter too, but in that case it seems more obvious that you would explicitly fail since they are the same node

/cc @chunhtai @pdblasi-google

one thing that may make this hard is that there may be internal semantics node wrapping each other, for example, the text widget may be wrapped with other semantics node before it is attached to the semantics node of the list view. We would have to figure out how to handle this.

API wise, I would lean towards the list parameter myself. Two reasons for it from my perspective:

1. It's more readable (though this is subjective)
2. Implementation wise, it gives us the opportunity to check the order in one or two passes instead of having to traverse the tree for each check.

For developer experience sake, I'd take two passes. One to confirm that all of the expected elements are there at all, then another pass to validate the order. That way it's simple to distinguish between a missing element, or unordered elements.

@chunhtai could you expand on the wrapping case you're talking about? I think I know what you're saying, but I want to be sure.

In this case both the First and Name finder would correspond to the same semantics node. Does this pass? or fail? or just add a warning to the logs?

I think I would fail in that case since First and Name are part of the same node so you can't make assertions about node ordering between them.

hasSemanticsElements

We should pick a name that implies that this matcher is about ordering.

In this case both the First and Name finder would correspond to the same semantics node. Does this pass? or fail? or just add a warning to the logs?

I think I would fail in that case since First and Name are part of the same node so you can't make assertions about node ordering between them.

I agree that this would be a failure case. We're checking semantic elements, not necessarily individual widgets. So if they're combined semantically, but the user is trying to check via the widgets that were created, then that's an error in the writing of the test.

hasSemanticsElements

We should pick a name that implies that this matcher is about ordering.

I agree with this, explicit is always better. I was thinking something along the lines of hasOrderedSemanticsElements or hasSemanticsElementsInOrder. Alternatively, we could have hasSemanticsElements have an optional boolean parameter ordered and have it capable of both and ordered and unordered validation.

One more note: With semantics nodes, "order" is a little ambiguous because there is hit test order and traversal order. I don't think we have to clear this up in the matcher's name, but the accompanying doc should be very specific about what order we're talking about. (And I think this issue is only talking about traversal order, @bryanoltman?)

for example if you have a scaffold

Scaffold(
          appBar: AppBar(title: const Text('Flutter Code Sample')),
          body: Center(
            child: Column(
              mainAxisAlignment: MainAxisAlignment.center,
              children: const <Widget>[
                Text('Row 1'),
                Text('Row 2'),
                Text('Row 3'),
              ],
            ),
          ),
        )

If you look at the semantics tree, you will notice there is an internal semantics node that wraps the appbar 'Flutter Code Sample'

In real life there may be multiple level of wrapping between the child and the parent depends on the structure of the widget tree.

There is also possibility that the parent itself may create multiple level of wrapping.

(And I think this issue is only talking about traversal order, @bryanoltman?)

Yep!

So, I'm going to try to give a summary of the known requirements as I see them at the moment, just to make sure everyone's on the same page (or at least in the same book). I'll summarize open questions in a followup comment.

1. The order will be checked against the traversal order of the semantics nodes
   • @chunhtai, I believe this will handle your nesting edge case by virtue of effectively flattening the tree into an iterable list of nodes
2. The matcher created (name TBD) will:
   • Take a list of finders that define the expected semantic elements
     • This will likely make use of the finder created in 107859 for a better developer experience
   • First validate that all expected semantic elements exist in the tree
   • Then validate that those elements are in the expected order

Here are the open questions that I still see:

1. What do we name this new matcher? Currently proposed names are:
   • hasSemanticsElements
     • Personally, I think this isn't explicit enough unless checking against order is optional via a parameter
   • hasOrderedSemanticsElements
   • hasSemanticsElementsInOrder
2. Do we want to support unordered validations?
   • I don't have a specific use case in mind for unordered semantics validation, as the traversal order is the primary validation I can think of that'd need to check more than one SemanticsElement at one time

Regarding 1): I would throw hasOrderedSemantics into the ring :) I don't think the Element in the name buys us much and might get confused with Flutter's element concept.

Regarding 2): Since I also can't think of a great use case for unordered assertions, I would not provide such a matcher for now - until we have a valid use case.

Oooh. I like hasOrderedSemantics. Shorter, but still explicit. Good idea.

I like hasOrderedSemantics! A couple questions:

1. Would incomplete but correctly ordered lists pass or fail? For example, using the address input form example, if I were to write:

expect(
  find.byKey(ValueKey('myForm')),
  hasOrderedSemantics([
    find.text('First Name'),
    find.text('Address 2'),
    find.text('ZIP'),
  ]),
);

Would that pass? I don't know that I feel strongly either way, but I suppose I'd lean towards this failing as I'm not super convinced there's a use case for a partial list.

2. What are we matching hasOrderedSemantics against? i.e., what is the first argument to expect?

3. Do we need to make a distinction between the semantics of a single element vs the list of semantics elements that the screen reader would traverse? This might be obvious based on the answer to the previous question.

Would that pass? I don't know that I feel strongly either way, but I suppose I'd lean towards this failing as I'm not super convinced there's a use case for a partial list.

I dont think figure out the complete traversal list is possible. There is the internal semantics node that convey information but a11y unfocusable

for example,

Scaffold(
      appBar: AppBar(title: const Text('Oops')),
      body: Center(
        child: Column(
          // index: 0,
          children: [
            Semantics(scopesRoute: true, label: 'hello', explicitChildNodes: true, child: Placeholder(),),
            ElevatedButton(child:Text('click'),onPressed: (){
              SemanticsService.announce("flutter accessibility announcement", TextDirection.ltr);
            })
          ],
        ),
      ),
    );

We also cannot tell whether a semantics node is a11y focusable from the framework side.

The other issue we may run into is the additional semantics node in a listview due to cache extent.

1. Would incomplete but correctly ordered lists pass or fail? For example, using the address input form example, if I were to write:

I was thinking that this would pass. Specifically for making it easy to split up tests. For example you could want a test that all of the labels for your form were in order, then you may want a separate test for validating that after you type something in for "First Name", it reads directly after the "First Name" label.

2. What are we matching hasOrderedSemantics against? i.e., what is the first argument to expect?

3. Do we need to make a distinction between the semantics of a single element vs the list of semantics elements that the screen reader would traverse? This might be obvious based on the answer to the previous question.

My assumption here was that the first argument was the top level that you'd be checking against, so you would expect all semantics elements being checked against to be children of the first argument of the expect.

That said, I was also confused when I originally looked at the proposed syntax. I don't have any better ideas in mind though.

It may be worth revisiting the syntax such that the finder should gather all of the relevant elements, and the matcher only validates the order of the gathered elements. That could be useful as a more generic tool as well, instead of just being a tool for semantics.

@chunhtai @goderbauer @jonahwilliams @bryanoltman

I've got some updates for this ticket! Apologies, I need to get better at providing in progress updates.

As @chunhtai mentioned, the platforms have slightly different implementations saying what is important for semantics and when. Because of this, I went with a simulatedTraversal method which makes a best effort traversal that should be useful for testing, but may differ from the platforms on edge cases. This method was added to a new SemanticsController, which I plan on expanding further with #112413.

I ended up going with a more general inOrder matcher that makes use of other matchers to verify order. This will also allow us to validate non-semantic things, such as validating the sorting of a list of Cards or similar use cases.

I think the end result ended up pretty readable and clean, see this example:

expect(tester.semantics.simulatedTraversal(), inOrder(strict: true, <Matcher>[
  containsSemantics(isHeader: true, label: 'Semantics Test'),
  containsSemantics(isTextField: true),
  containsSemantics(label: 'Off Switch'),
  containsSemantics(hasToggledState: true),
  containsSemantics(label: 'On Switch'),
  containsSemantics(hasToggledState: true, isToggled: true),
  containsSemantics(label: "Multiline\nIt's a\nmultiline label!"),
  containsSemantics(label: 'Slider'),
  containsSemantics(isSlider: true, value: '50%'),
  containsSemantics(label: 'Enabled Button'),
  containsSemantics(isButton: true, label: 'Tap'),
  containsSemantics(label: 'Disabled Button'),
  containsSemantics(isButton: true, label: "Don't Tap"),
  containsSemantics(label: 'Checked Radio'),
  containsSemantics(hasCheckedState: true, isChecked: true),
  containsSemantics(label: 'Unchecked Radio'),
  containsSemantics(hasCheckedState: true, isChecked: false),
]));

In the example given above I was slightly surprised to see tester.semantics.simulatedTraversal() as the first argument to expect. I think I would have expected to find some kind of a widget matcher there to describe the part of the UI I want to verify the order for.

Does this setup mean, I always have to assert order globally on the entire app under test? Or can I somehow express that all I care about is that button X is traversed directly before button Y? Without asserting that button X must come before text Foo?

Well, the usual order of the expect call for widget testing is expect(<finder>, <matcher>), so normally I would expect to see a Finder in the first argument. The issue with using a Finder is that the Finder API is based off of Elements, and the SemanticsNodes used in the semantics tree are not Elements.

Because of those limitations, I went with the new SemanticsController and a method that returns a list of SemanticsNodes directly instead of trying to force the Finder API to work in ways it wasn't meant to.

That said, the simulatedTraversal does provide a start parameter to allow you to limit the returned list somewhat. I'm thinking of adding in an end parameter as well to further limit the returned items.

The inOrder matcher as it is defaults to searching fuzzily, so it would allow you to verify just the order of a few items if that's all you needed, with the strict parameter forcing it to validate against all items. As you mentioned in the PR, there may be other existing methods that allow the fuzzy/strict match as well, so the inOrder matcher may go away.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.