add: Parse `Scenario` as a synonym for `Scenario Outline` #316

acoulton · 2025-05-06T16:26:06Z

In cucumber/gherkin, the Scenario: and Scenario Outline: keywords are interchangeable. Either may have (or not have) an Examples: table.

We still need to differentiate between OutlineNode and ScenarioNode in the parsed result, for BC and because Behat treats these differently at runtime.

However, we can safely parse a Scenario: that contains an Examples: table (which would have previously been invalid) to an OutlineNode - exactly as if the user had written Scenario Outline:.

This does require refactoring how we parse tags, because:

Previously, we could make assumptions based on whether we were expecting (or had already seen) an Examples table.
Now, we can't tell from the tags alone whether they belong to an Examples table, or mark the end of the current Scenario and the start of the next.

Therefore, we now scan ahead to predict which node the tags are attached to. This also allows us to throw a clearer exception if tags are found in unexpected places in the document.

Relates to #153

In cucumber/gherkin, the `Scenario:` and `Scenario Outline:` keywords are interchangeable. Either may have (or not have) an `Examples:` table. We still need to differentiate between `OutlineNode` and `ScenarioNode` in the parsed result, for BC and because Behat treats these differently at runtime. However, we can safely parse a `Scenario:` that contains an `Examples:` table (which would have previously been invalid) to an `OutlineNode` - exactly as if the user had written `Scenario Outline:`. This does require refactoring how we parse tags, because: * Previously, we could make assumptions based on whether we were expecting (or had already seen) an Examples table. * Now, we can't tell from the tags alone whether they belong to an Examples table, or mark the end of the current Scenario and the start of the next. Therefore, we now scan ahead to predict which node the tags are attached to. This also allows us to throw a clearer exception if tags are found in unexpected places in the document.

codecov · 2025-05-06T16:26:45Z

Codecov Report

Attention: Patch coverage is 98.41270% with 1 line in your changes missing coverage. Please review.

Project coverage is 96.70%. Comparing base (7eaf0ec) to head (0816d7e).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
src/Parser.php	97.87%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #316      +/-   ##
============================================
+ Coverage     96.23%   96.70%   +0.47%     
- Complexity      598      602       +4     
============================================
  Files            35       36       +1     
  Lines          1751     1760       +9     
============================================
+ Hits           1685     1702      +17     
+ Misses           66       58       -8

Flag	Coverage Δ
php8.1	`96.70% <98.41%> (+0.47%)`	⬆️
php8.1--with=symfony/yaml:^5.4	`96.70% <98.41%> (+0.47%)`	⬆️
php8.1--with=symfony/yaml:^6.4	`96.70% <98.41%> (+0.47%)`	⬆️
php8.2	`96.70% <98.41%> (+0.47%)`	⬆️
php8.3	`96.70% <98.41%> (+0.47%)`	⬆️
php8.4	`96.70% <98.41%> (+0.47%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

acoulton · 2025-05-06T16:27:14Z

src/Parser.php

    private $tags = [];
    private $languageSpecifierLine;

-    private $passedNodesStack = [];


This was only used to track recursion into nodes for the purpose of making assumptions about whether we could take tags (and looks like it wasn't always unwound). This is now handled differently.

acoulton · 2025-05-06T16:28:38Z

src/Parser.php

+            // Should not be possible to happen, parseTags should have already picked this up.
+            throw new UnexpectedTaggedNodeException($token, $this->file);


I have kept this for the moment out of caution, I don't think it can happen now (it doesn't with any of the test examples as Codecov has also detected).

acoulton · 2025-05-06T16:29:42Z

src/Parser.php

+        if (!($node instanceof OutlineNode && $node->hasExamples())) {
+            throw new ParserException(sprintf(
+                'Outline should have examples table, but got none for outline "%s" on line: %d%s',
+                $node->getTitle(),
+                $node->getLine(),
+                $this->file ? ' in file: ' . $this->file : ''
+            ));
+        }


NB cucumber/gherkin doesn't throw in this case (see #153) - I have kept it for now but plan to look at that separately.

I think solving #153 fully is just a matter of removing that exception, by fully making them synonyms.

I agree, I just mean it's a separate PR/change. It relates to other cases where we currently throw on partially-complete feature files that gherkin accepts and I think we should consider that in the round.

well, to me, accepting outlines without examples (returning them as a ScenarioNode) is part of making Scenario and Scenario Outline be synonyms. Right now, you are making Scenario a synonym of Scenario Outline but Scenario Outline is not a synonym of Scenario, which is quite weird. Synonyms are meant to be bidirectional relations to me.

Right now, you are making Scenario a synonym of Scenario Outline but Scenario Outline is not a synonym of Scenario

Which is why I was careful to title the PR "add: Parse Scenario as a synonym for Scenario Outline" and not "Treat Scenario and Scenario Outlines as synonyms" 😆

Synonyms are meant to be bidirectional relations to me.

Synonyms are very often only synonyms in context, I can think of lots of examples of synonyms that are not always bidirectional, or where one is a generalisation of a more specific term.

A Scenario with an Examples table is clearly equivalent to a Scenario Outline. Is a Scenario Outline (where the writer has taken care to use the specific term) without an Examples table always the same as a Scenario? I think that's open to debate (but of course we'll eventually follow cucumber/gherkin's opinion).

The bigger point is - I am trying to move forward with cucumber/gherkin parity (which has been stalled for years). Some things - like this PR - are very obviously just new features with no BC impact.

Others - like no longer throwing if the user said "Scenario Outline" but did not provide a table - involve changes to existing behaviour that is currently explicitly tested.

Therefore I want to split these changes up into separate commits / PRs so that we can properly review them and consider any BC impact. I do not want to ship another release that breaks people's CI because we overlooked something, so I think we should keep each change as incremental as possible.

I note that I said from the moment I opened the PR that I would be coming back to this specific point (which we also have an open issue for) so tbh I don't see why it should block approving / merging these changes.

I would say that turning a parsing exception into a supported case won't break CI of anyone: if they have files matching that case, it will potentially fix their CI. Turning invalid files into valid ones is generally not considered a BC break (otherwise, it would be impossible to add any new syntax)
And this is also exactly the kind of changes you are doing for files using Scenario that have an Examples table today (which are also a ParserException).

But anyway, I won't block the merge on that if you decide to keep it split.

Fair points and broadly I agree, I just think there's a philosophical difference between changing behaviour that is undefined, and changing behaviour that we have explicit tests for. Hence I prefer the discipline of considering the second type of changes separately. The next PR will be very soon 😄

acoulton · 2025-05-06T16:32:51Z

src/Parser.php

+        if ($examples !== []) {
+            return new OutlineNode(rtrim($title) ?: null, $tags, $steps, $examples, $keyword, $line);
        }

-        return new OutlineNode(rtrim($title) ?: null, $tags, $steps, $examples, $keyword, $line);
+        return new ScenarioNode(rtrim($title) ?: null, $tags, $steps, $keyword, $line);


Note that at the moment calling ->getKeyword() on the parsed node returns a hardcoded result. So if the feature file used Scenario: but had an examples table then e.g. Behat's pretty printer would still print Scenario Outline:.

We could add a new property into the nodes to record the actual keyword that we found, but I would be a little nervous about BC impact if the result of OutlineNode->getKeyword() can vary in case anyone is matching on / doing anything specific with that string.

acoulton · 2025-05-06T16:34:22Z

tests/ParserExceptionsTest.php


        $this->expectExceptionObject(
-            new ParserException('Expected Scenario, Outline or Background, but got Step on line: 6')
+            new ParserException('Step can not be tagged, but it is on line: 6')


I think this new exception is clearer than the previous anyway. I have left the assertion as ParserException (rather than change to the actual UnexpectedTaggedNodeException) to prove that this is all still BC.

acoulton · 2025-05-06T16:35:09Z

tests/ParserTest.php

        } catch (ParserException $e) {
            // expected - features cannot end with tags
-            $this->assertSame('Unexpected end of file after tags on line 5', $e->getMessage());
+            $this->assertSame('Unexpected end of file after tags on line: 5', $e->getMessage());


Slight change to the format for consistency with the other exceptions.

carlos-granados

Approved, just a tiny comment, feel free to ignore it

src/Parser.php

stof

should we throw a ParserException if we find steps after an examples table ? currently, the parser seems to allow mixing them AFAICT (unless the token type determination returns a different result for such case).

stof · 2025-05-13T11:12:47Z

tests/Cucumber/CompatibilityTest.php

        'descriptions_with_comments.feature' => 'Examples table descriptions not supported',
        'extra_table_content.feature' => 'Table without right border triggers a ParserException',
        'incomplete_scenario_outline.feature' => 'Scenario and Scenario outline not yet synonyms',
        'padded_example.feature' => 'Scenario and Scenario outline not yet synonyms',


what about those 2 that were skipped for the same reason ?

They now fail for unrelated reasons.

incomplete_scenario_outline.feature because we throw if an Outline has no Examples table, but cucumber/gherkin does not (tracked in Scenario Outlines inconsistencies #153 and noted in the diff above).

padded_example.feature because we don't trim table padding (specifically I think) the same as cucumber/gherkin - no issue for this yet (I plan to create one), and it has BC implications.

I'll update the reason descriptions in the testcase

acoulton · 2025-05-13T14:03:44Z

should we throw a ParserException if we find steps after an examples table?

Yeah, probably - it does seem that (as now) we don't detect / fail in that situation. I'll double check what cucumber/gherkin does.

We were not previously enforcing that an Examples: table can only be followed by more Examples: tables, comments, whitespace or the end of the Scenario / file. With this commit, that will now cause an UnexpectedParserNodeException. I have refactored the order of checking whether we can accept a particular node when parsing a scenario / outline body to make the logic easier. It also slightly changes the message of the exception for existing invalid cases so that they are more accurate about what is expected in the current state.

acoulton · 2025-05-14T11:58:19Z

I checked and @cucumber/gherkin does indeed throw a Parser error if Steps follow Examples: - https://codesandbox.io/p/devbox/gherkin-parser-ast-forked-f9j44t?workspaceId=ws_Mrsa3f4At4n3w9MazShoyf

I have added a test for that and updated the Parser to detect and throw in this situation.

stof · 2025-05-14T12:20:19Z

I have added a test for that and updated the Parser to detect and throw in this situation.

I suggest contributing such a test for bad data to the cucumber testsuite as well, to make it part of the shared tests in the future.

carlos-granados

@acoulton looking great overall, just a small comment

carlos-granados · 2025-05-16T06:46:46Z

src/Parser.php

+     *
+     * @throws UnexpectedTaggedNodeException if there is not a taggable node
+     */
+    private function predictNextTaggedNodeType(): string


This function actually does two things: validates that the next tagged node is correct and returns it. But the function name does not reflect the validation part, so when we call it and don't use the return value it looks weird. Also predict sounds a bit magic in my head 😆 . What about validateAndGetNextTaggedNodeType()?

acoulton · 2025-05-16T07:49:23Z

@carlos-granados good catch, thanks. The method started off just looking ahead, and I named it predict to follow the naming of other methods in this class & the Lexer. But then moved the validation & exception inside the method and didn't rename it. Fixed now.

carlos-granados

All good now

acoulton requested a review from carlos-granados May 6, 2025 16:26

acoulton commented May 6, 2025

View reviewed changes

carlos-granados approved these changes May 7, 2025

View reviewed changes

src/Parser.php Outdated Show resolved Hide resolved

stof reviewed May 13, 2025

View reviewed changes

nits: Code review fixes

54bc501

acoulton force-pushed the feat-scenario-outline-synonym branch from c8307f7 to ead1ec8 Compare May 14, 2025 11:51

acoulton force-pushed the feat-scenario-outline-synonym branch from ead1ec8 to 2a43d41 Compare May 14, 2025 11:57

acoulton requested review from carlos-granados and stof May 14, 2025 11:58

carlos-granados reviewed May 16, 2025

View reviewed changes

style: Rename method for clarity

0816d7e

acoulton requested a review from carlos-granados May 16, 2025 07:49

carlos-granados approved these changes May 16, 2025

View reviewed changes

acoulton merged commit 8b80bff into Behat:master May 16, 2025
10 checks passed

acoulton deleted the feat-scenario-outline-synonym branch May 16, 2025 09:50

acoulton mentioned this pull request May 16, 2025

Parse Scenario Outline with no examples as a Scenario #324

Merged

		// Should not be possible to happen, parseTags should have already picked this up.
		throw new UnexpectedTaggedNodeException($token, $this->file);

add: Parse Scenario as a synonym for Scenario Outline #316

add: Parse Scenario as a synonym for Scenario Outline #316

Uh oh!

Conversation

acoulton commented May 6, 2025

Uh oh!

codecov bot commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

acoulton May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlos-granados left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stof left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

acoulton commented May 13, 2025

Uh oh!

acoulton commented May 14, 2025

Uh oh!

stof commented May 14, 2025

Uh oh!

carlos-granados left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

acoulton commented May 16, 2025

Uh oh!

carlos-granados left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

add: Parse `Scenario` as a synonym for `Scenario Outline` #316

add: Parse `Scenario` as a synonym for `Scenario Outline` #316

codecov bot commented May 6, 2025 •

edited

Loading

acoulton May 6, 2025 •

edited

Loading