PREG_BACKTRACK_LIMIT_ERROR in Yaml parser #20411

RichardBradley · 2016-11-04T16:34:25Z

A YAML string with too many backslash escapes can trigger a PREG_BACKTRACK_LIMIT_ERROR error in the Yaml parser.

Steps to reproduce:

Run the following unit test:

use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;
use Symfony\Component\Yaml\Yaml;

class YamlParserBacktrackLimitTest extends WebTestCase
{
    public function testYamlQuotedString() {

        $foo = str_repeat("x\r\n\\\"x\"x", 1000);
        $yamlString = Yaml::dump(["foo" => $foo]);

        $arrayFromYaml = Yaml::parse($yamlString);

        $this->assertEquals($foo, $arrayFromYaml['foo']);
    }
}

Observed behaviour

This test will pass if you reduce the "1000" constant to "100", and fail as-is.

The reported exception is:

Malformed inline YAML string ("x\r\n\\\ ...
at n/a
        in ...\vendor\symfony\symfony\src\Symfony\Component\Yaml\Inline.php line 344

The relevant source is here:

       const REGEX_QUOTED_STRING = '(?:"([^"\\\\]*(?:\\\\.[^"\\\\]*)*)"|\'([^\']*(?:\'\'[^\']*)*)\')';
...
        if (!preg_match('/'.self::REGEX_QUOTED_STRING.'/Au', substr($scalar, $i), $match)) {
            throw new ParseException(sprintf('Malformed inline YAML string: %s.', substr($scalar, $i)));
        }

Expected behaviour

The test should pass, i.e. the YAML should be parseable, since it fits in memory and is well formed.
The exception thrown by Symfony if the preg_match fails due to PREG_BACKTRACK_LIMIT_ERROR should indicate that the op failed due to resource exhaustion and not incorrectly blame invalid YAML.

Suggested fix

I suspect that changing the REGEX_QUOTED_STRING regex to use possessive qualifiers will probably fix bug 1 above without otherwise changing the YAML parser's behaviour.

Fixing bug 2 above needs a call to preg_last_error() after each "preg_match" call in Symfony code, which will be tedious.

The text was updated successfully, but these errors were encountered:

stof · 2016-11-04T16:47:08Z

using possessive quantifiers should indeed be done.

Regarding your point 2, this would involve distinguishing 0 and false as return value for preg_match (false would indicate a regex error, which should trigger a different exception saying that something was broken in the parser).
But this is quite tedious indeed, so it would be better to fix the root error (avoid needless backtracking)

…ardBradley) This PR was merged into the 2.7 branch. Discussion ---------- #20411 fix Yaml parsing for very long quoted strings | Q | A | ------------- | --- | Branch? | 2.7 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #20411 | License | MIT | Doc PR | no This fixes #20411, a YAML string with too many backslash escapes can trigger a `PREG_BACKTRACK_LIMIT_ERROR` error in the Yaml parser. There should be no behavioural change other than the bug fix I have included a test which fails before this fix and passes after this fix. Commits ------- 51bca66 #20411 fix Yaml parsing for very long quoted strings

mahagr · 2017-01-14T08:07:01Z

👍 Thanks for fixing this one, its a big one. It should also improve yaml parsing performance (on affected input) by a great deal.

RichardBradley · 2017-01-14T11:52:06Z

You're welcome.

Do you know when I can expect a 3.2 based release which will include this fix?

mahagr · 2017-01-14T13:23:16Z

I'm waiting for the release as much as you do :)

xabbuh · 2017-01-14T17:37:33Z

Since the last patch releases have just been released two days ago you will have to wait for about four weeks for the next releases.

* 2.7: fixed typo Revert "fixed typo" fixed typo fixed CS Avoid setting request attributes from signature arguments in AnnotationClassLoader [DependencyInjection] Add some missing typehints in YamlFileLoader [DependencyInjection] minor: Fix a DocBlock [HttpKernel] Give higher priority to adding request formats [FrameworkBundle] Fix third level headers for MarkdownDescriptor [TwigBundle] do not lose already set method calls #20411 fix Yaml parsing for very long quoted strings CS: apply is_null DX: remove invalid inheritdoc bumped Symfony version to 2.7.24 updated VERSION for 2.7.23 update CONTRIBUTORS for 2.7.23 updated CHANGELOG for 2.7.23 [FrameworkBundle] Skip test if xdebug.file_link_format is defined.

* 2.8: (26 commits) fixed CS fixed CS fixed CS fixer config fixed typo Revert "fixed typo" fixed typo fixed CS Avoid setting request attributes from signature arguments in AnnotationClassLoader [DependencyInjection] Add some missing typehints in YamlFileLoader [DependencyInjection] minor: Fix a DocBlock [HttpKernel] Give higher priority to adding request formats [PropertyInfo] Don't try to access a property thru a static method [PropertyInfo] Exclude static methods form properties guessing [FrameworkBundle] Fix third level headers for MarkdownDescriptor [TwigBundle] do not lose already set method calls #20411 fix Yaml parsing for very long quoted strings CS: apply is_null DX: remove invalid inheritdoc bumped Symfony version to 2.8.17 updated VERSION for 2.8.16 ...

* 3.1: (31 commits) fixed CS fixed CS fixed CS fixer config fixed typo Revert "fixed typo" fixed typo fixed CS Avoid setting request attributes from signature arguments in AnnotationClassLoader [DependencyInjection] Add some missing typehints in YamlFileLoader [DependencyInjection] minor: Fix a DocBlock [HttpKernel] Give higher priority to adding request formats [PropertyInfo] Don't try to access a property thru a static method [PropertyInfo] Exclude static methods form properties guessing [FrameworkBundle] Fix third level headers for MarkdownDescriptor [Ldap] Using Ldap stored username instead of form submitted one [Ldap] load users with the good username case [DoctrineBridge] Fixed invalid unique value as composite key [Doctrine Bridge] fix UniqueEntityValidator for composite object primary keys [TwigBundle] do not lose already set method calls #20411 fix Yaml parsing for very long quoted strings ...

…ardBradley) This PR was squashed before being merged into the 2.7 branch (closes #21523). Discussion ---------- #20411 fix Yaml parsing for very long quoted strings | Q | A | ------------- | --- | Branch? | 2.7 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #20411 | License | MIT | Doc PR | no This is a second fix for the issue discussed in #20411. My first PR (#21279) didn't fix the bug in all cases, sorry. If a YAML string has too many spaces in the value, it can trigger a `PREG_BACKTRACK_LIMIT_ERROR` error in the Yaml parser. There should be no behavioural change other than the bug fix I have included a test which fails before this fix and passes after this fix. I have also added checks that detect other PCRE internal errors and throw a more descriptive exception. Before this patch, the YAML engine would often give incorrect results, rather than throwing, on a PCRE `PREG_BACKTRACK_LIMIT_ERROR` error. Commits ------- c9a1c09 #20411 fix Yaml parsing for very long quoted strings

* 2.7: #20411 fix Yaml parsing for very long quoted strings [Doctrine Bridge] fix priority for doctrine event listeners Use PHP functions as array_map callbacks when possible [Validator] revert wrong Phpdoc change Use proper line endings

* 2.8: [DI] Fix PhpDumper generated doc block #20411 fix Yaml parsing for very long quoted strings [Doctrine Bridge] fix priority for doctrine event listeners Use PHP functions as array_map callbacks when possible [Validator] revert wrong Phpdoc change Use proper line endings

* 3.2: [Yaml] CS [DI] Fix PhpDumper generated doc block #20411 fix Yaml parsing for very long quoted strings [Workflow] add Phpdoc for better IDE support fix package name in conflict rule improve message when workflows are missing [Doctrine Bridge] fix priority for doctrine event listeners Use PHP functions as array_map callbacks when possible [Validator] revert wrong Phpdoc change Use proper line endings

javiereguiluz added the Yaml label Nov 5, 2016

RichardBradley added a commit to RichardBradley/symfony that referenced this issue Jan 13, 2017

symfony#20411 fix Yaml parsing for very long quoted strings

e464160

RichardBradley added a commit to RichardBradley/symfony that referenced this issue Jan 13, 2017

symfony#20411 fix Yaml parsing for very long quoted strings

100a0ce

RichardBradley mentioned this issue Jan 13, 2017

#20411 fix Yaml parsing for very long quoted strings #21279

Merged

RichardBradley added a commit to RichardBradley/symfony that referenced this issue Jan 13, 2017

symfony#20411 fix Yaml parsing for very long quoted strings

51bca66

xabbuh closed this as completed Jan 14, 2017

xabbuh added the Bug label Jan 14, 2017

carsonbot added the Status: Needs Review label Jan 14, 2017

xabbuh removed the Status: Needs Review label Jan 14, 2017

xabbuh mentioned this issue Jan 14, 2017

[yaml] Inline::parseQuotedScalar() causes stack overflow in Windows is field contains a lot of double quotes #14722

Closed

fabpot mentioned this issue Jan 28, 2017

Release v3.1.10 #21441

Merged

RichardBradley added a commit to RichardBradley/symfony that referenced this issue Feb 3, 2017

symfony#20411 fix Yaml parsing for very long quoted strings

6e001c8

RichardBradley mentioned this issue Feb 3, 2017

#20411 fix Yaml parsing for very long quoted strings #21523

Closed

This was referenced Feb 6, 2017

Release v2.7.24 #21541

Merged

Release v2.8.17 #21543

Merged

Release v3.2.3 #21544

Merged

RichardBradley added a commit to RichardBradley/symfony that referenced this issue Mar 1, 2017

symfony#20411 fix Yaml parsing for very long quoted strings

ec7f68e

RichardBradley added a commit to RichardBradley/symfony that referenced this issue Mar 1, 2017

symfony#20411 fix Yaml parsing for very long quoted strings

0152ad9

RichardBradley added a commit to RichardBradley/symfony that referenced this issue Mar 10, 2017

symfony#20411 reformat conditional as requested

757364d

fabpot pushed a commit that referenced this issue Mar 17, 2017

#20411 fix Yaml parsing for very long quoted strings

c9a1c09

This was referenced Apr 4, 2017

Release v2.7.26 #22263

Merged

Release v2.8.19 #22284

Merged

Release v3.2.7 #22293

Merged

MatTheCat mentioned this issue Feb 10, 2020

PCRE backtrack-limit hit on huge HTML jolicode/seo-override#43

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PREG_BACKTRACK_LIMIT_ERROR in Yaml parser #20411

PREG_BACKTRACK_LIMIT_ERROR in Yaml parser #20411

RichardBradley commented Nov 4, 2016 •

edited by javiereguiluz

Loading

stof commented Nov 4, 2016

mahagr commented Jan 14, 2017

RichardBradley commented Jan 14, 2017

mahagr commented Jan 14, 2017

xabbuh commented Jan 14, 2017

PREG_BACKTRACK_LIMIT_ERROR in Yaml parser #20411

PREG_BACKTRACK_LIMIT_ERROR in Yaml parser #20411

Comments

RichardBradley commented Nov 4, 2016 • edited by javiereguiluz Loading

Steps to reproduce:

Observed behaviour

Expected behaviour

Suggested fix

stof commented Nov 4, 2016

mahagr commented Jan 14, 2017

RichardBradley commented Jan 14, 2017

mahagr commented Jan 14, 2017

xabbuh commented Jan 14, 2017

RichardBradley commented Nov 4, 2016 •

edited by javiereguiluz

Loading