Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-135462: Fix quadratic complexity in processing special input in HTMLParser #135464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 13, 2025
+111 −31
Merged

gh-135462: Fix quadratic complexity in processing special input in HTMLParser #135464

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c87cb49
gh-135462: Fix quadratic complexity in processing special input in HT…
serhiy-storchaka Jun 13, 2025
fe9730d
Apply suggestions from code review
serhiy-storchaka Jun 13, 2025
5fed8ef
Update a NEWS entry.
serhiy-storchaka Jun 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 30 additions & 11 deletions Lib/html/parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
attr_charref = re.compile(r'&(#[0-9]+|#[xX][0-9a-fA-F]+|[a-zA-Z][a-zA-Z0-9]*)[;=]?')

starttagopen = re.compile('<[a-zA-Z]')
endtagopen = re.compile('</[a-zA-Z]')
piclose = re.compile('>')
commentclose = re.compile(r'--\s*>')
# Note:
Expand Down Expand Up @@ -195,25 +196,43 @@ def goahead(self, end):
k = self.parse_pi(i)
elif startswith("<!", i):
k = self.parse_html_declaration(i)
elif (i + 1) < n:
elif (i + 1) < n or end:
self.handle_data("<")
k = i + 1
else:
break
if k < 0:
if not end:
break
k = rawdata.find('>', i + 1)
if k < 0:
k = rawdata.find('<', i + 1)
if k < 0:
k = i + 1
else:
k += 1
if self.convert_charrefs and not self.cdata_elem:
self.handle_data(unescape(rawdata[i:k]))
if starttagopen.match(rawdata, i): # < + letter
pass
elif startswith("</", i):
if i + 2 == n:
self.handle_data("</")
elif endtagopen.match(rawdata, i): # </ + letter
pass
else:
# bogus comment
self.handle_comment(rawdata[i+2:])
elif startswith("<!--", i):
j = n
for suffix in ("--!", "--", "-"):
if rawdata.endswith(suffix, i+4):
j -= len(suffix)
break
self.handle_comment(rawdata[i+4:j])
elif startswith("<![CDATA[", i):
self.unknown_decl(rawdata[i+3:])
elif rawdata[i:i+9].lower() == '<!doctype':
self.handle_decl(rawdata[i+2:])
elif startswith("<!", i):
# bogus comment
self.handle_comment(rawdata[i+2:])
elif startswith("<?", i):
self.handle_pi(rawdata[i+2:])
else:
self.handle_data(rawdata[i:k])
raise AssertionError("we should not get here!")
k = n
i = self.updatepos(i, k)
elif startswith("&#", i):
match = charref.match(rawdata, i)
Expand Down
97 changes: 77 additions & 20 deletions Lib/test/test_htmlparser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import unittest

from unittest.mock import patch
from test import support


class EventCollector(html.parser.HTMLParser):
Expand Down Expand Up @@ -430,28 +431,34 @@ def test_tolerant_parsing(self):
('data', '<'),
('starttag', 'bc<', [('a', None)]),
('endtag', 'html'),
('data', '\n<img src="URL>'),
('comment', '/img'),
('endtag', 'html<')])
('data', '\n')])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that now everything after the first </html> is ignored (except the \n). This is technically a change in behavior, which should be fine if the new behavior matches the HTML5 specs, but maybe should be noted in the whatsnew.

There also seem to be other minor changes in behavior that -- if they follow the specs -- might not need to be documented (a generic "Some additional invalid constructs are now handled according to the HTML5 specs." might be enough)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, a double-quoted attribute value is never closed. This is https://html.spec.whatwg.org/multipage/parsing.html#parse-error-eof-in-tag .

I have update the NEWS entry.


def test_starttag_junk_chars(self):
self._run_check("<", [('data', '<')])
self._run_check("<>", [('data', '<>')])
self._run_check("< >", [('data', '< >')])
self._run_check("< ", [('data', '< ')])
self._run_check("</>", [])
self._run_check("<$>", [('data', '<$>')])
self._run_check("</$>", [('comment', '$')])
self._run_check("</", [('data', '</')])
self._run_check("</a", [('data', '</a')])
self._run_check("</a", [])
self._run_check("</ a>", [('endtag', 'a')])
self._run_check("</ a", [('comment', ' a')])
self._run_check("<a<a>", [('starttag', 'a<a', [])])
self._run_check("</a<a>", [('endtag', 'a<a')])
self._run_check("<!", [('data', '<!')])
self._run_check("<a", [('data', '<a')])
self._run_check("<a foo='bar'", [('data', "<a foo='bar'")])
self._run_check("<a foo='bar", [('data', "<a foo='bar")])
self._run_check("<a foo='>'", [('data', "<a foo='>'")])
self._run_check("<a foo='>", [('data', "<a foo='>")])
self._run_check("<!", [('comment', '')])
self._run_check("<a", [])
self._run_check("<a foo='bar'", [])
self._run_check("<a foo='bar", [])
self._run_check("<a foo='>'", [])
self._run_check("<a foo='>", [])
self._run_check("<a$>", [('starttag', 'a$', [])])
self._run_check("<a$b>", [('starttag', 'a$b', [])])
self._run_check("<a$b/>", [('startendtag', 'a$b', [])])
self._run_check("<a$b >", [('starttag', 'a$b', [])])
self._run_check("<a$b />", [('startendtag', 'a$b', [])])
self._run_check("</a$b>", [('endtag', 'a$b')])

def test_slashes_in_starttag(self):
self._run_check('<a foo="var"/>', [('startendtag', 'a', [('foo', 'var')])])
Expand Down Expand Up @@ -576,21 +583,50 @@ def test_EOF_in_charref(self):
for html, expected in data:
self._run_check(html, expected)

def test_EOF_in_comments_or_decls(self):
def test_eof_in_comments(self):
data = [
('<!', [('data', '<!')]),
('<!-', [('data', '<!-')]),
('<!--', [('data', '<!--')]),
('<![', [('data', '<![')]),
('<![CDATA[', [('data', '<![CDATA[')]),
('<![CDATA[x', [('data', '<![CDATA[x')]),
('<!DOCTYPE', [('data', '<!DOCTYPE')]),
('<!DOCTYPE HTML', [('data', '<!DOCTYPE HTML')]),
('<!--', [('comment', '')]),
('<!---', [('comment', '')]),
('<!----', [('comment', '')]),
('<!-----', [('comment', '-')]),
('<!------', [('comment', '--')]),
('<!----!', [('comment', '')]),
('<!---!', [('comment', '-!')]),
('<!---!>', [('comment', '-!>')]),
('<!--foo', [('comment', 'foo')]),
('<!--foo-', [('comment', 'foo')]),
('<!--foo--', [('comment', 'foo')]),
('<!--foo--!', [('comment', 'foo')]),
('<!--<!--', [('comment', '<!')]),
('<!--<!--!', [('comment', '<!')]),
]
for html, expected in data:
self._run_check(html, expected)

def test_eof_in_declarations(self):
data = [
('<!', [('comment', '')]),
('<!-', [('comment', '-')]),
('<![', [('comment', '[')]),
('<![CDATA[', [('unknown decl', 'CDATA[')]),
('<![CDATA[x', [('unknown decl', 'CDATA[x')]),
('<![CDATA[x]', [('unknown decl', 'CDATA[x]')]),
('<![CDATA[x]]', [('unknown decl', 'CDATA[x]]')]),
('<!DOCTYPE', [('decl', 'DOCTYPE')]),
('<!DOCTYPE ', [('decl', 'DOCTYPE ')]),
('<!DOCTYPE html', [('decl', 'DOCTYPE html')]),
('<!DOCTYPE html ', [('decl', 'DOCTYPE html ')]),
('<!DOCTYPE html PUBLIC', [('decl', 'DOCTYPE html PUBLIC')]),
('<!DOCTYPE html PUBLIC "foo', [('decl', 'DOCTYPE html PUBLIC "foo')]),
('<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "foo',
[('decl', 'DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "foo')]),
]
for html, expected in data:
self._run_check(html, expected)

def test_bogus_comments(self):
html = ('<! not really a comment >'
html = ('<!ELEMENT br EMPTY>'
'<! not really a comment >'
'<! not a comment either -->'
'<! -- close enough -->'
'<!><!<-- this was an empty comment>'
Expand All @@ -604,6 +640,7 @@ def test_bogus_comments(self):
'<![CDATA]]>' # required '[' after CDATA
)
expected = [
('comment', 'ELEMENT br EMPTY'),
('comment', ' not really a comment '),
('comment', ' not a comment either --'),
('comment', ' -- close enough --'),
Expand Down Expand Up @@ -684,6 +721,26 @@ def test_convert_charrefs_dropped_text(self):
('endtag', 'a'), ('data', ' bar & baz')]
)

@support.requires_resource('cpu')
def test_eof_no_quadratic_complexity(self):
# Each of these examples used to take about an hour.
# Now they take a fraction of a second.
Comment on lines +724 to +727
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they now take a fraction of a second, is there a reason to require the cpu resource?

My understanding is that:

  • with the requires_resource('cpu') decorator:
    • this test would normally be skipped
    • in case of regression, we won't notice unless the cpu is enabled
  • without the decorator:
    • this test is always run and completes quickly
    • in case of regression, the test will timeout/fail and expose the problem

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They totally take 1.3 seconds on my computer. All other tests take 0.1-0.2 seconds. It is a waste of time to run it several times for every update of any PR. Some buildbots are slower than my computer.

I think that it is enough to run this test only on the fastests builtbots. We already used requires_resource('cpu') in similar tests.

def check(source):
parser = html.parser.HTMLParser()
parser.feed(source)
parser.close()
n = 120_000
check("<a " * n)
check("<a a=" * n)
check("</a " * 14 * n)
check("</a a=" * 11 * n)
check("<!--" * 4 * n)
check("<!" * 60 * n)
check("<?" * 19 * n)
check("</$" * 15 * n)
check("<![CDATA[" * 9 * n)
check("<!doctype" * 35 * n)


class AttributesTestCase(TestCaseBase):

Expand Down
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/Security/2025-06-13-15-55-22.gh-issue-135462.KBeJpc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Fix quadratic complexity in processing specially crafted input in
:class:`html.parser.HTMLParser`. End-of-file errors are now handled according
to the HTML5 specs -- comments and declarations are automatically closed,
tags are ignored.
Loading