Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-140797: Forbid capturing groups in re.Scanner lexicon patterns #140944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
serhiy-storchaka merged 4 commits into from
Nov 4, 2025
Merged

gh-140797: Forbid capturing groups in re.Scanner lexicon patterns #140944

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
29db6ca
This PR references issue gh-140797
Abhi210 Nov 3, 2025
feaee4e
gh-140797 forbid capturing groups in re.Scanner lexicon patterns- Rev-7
Abhi210 Nov 4, 2025
d1d582e
Update Lib/test/test_re.py
Abhi210 Nov 4, 2025
f4f7d4d
Update Lib/test/test_re.py
Abhi210 Nov 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion Lib/re/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -397,9 +397,16 @@ def __init__(self, lexicon, flags=0):
s = _parser.State()
s.flags = flags
for phrase, action in lexicon:
sub_pattern = _parser.parse(phrase, flags)
if sub_pattern.state.groups != 1: # <- 1 means always has \0
Comment thread
Abhi210 marked this conversation as resolved.
Outdated
raise ValueError(
Copy link
Copy Markdown

@wjssz wjssz Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can write in one line. A line should <= 80 characters.

raise ValueError("Can not use capturing groups in re.Scanner.")

Copy link
Copy Markdown
Contributor Author

@Abhi210 Abhi210 Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the suggestion. I have resolved it now

"re.Scanner lexicon patterns must not contain capturing groups;\n"
Comment thread
Abhi210 marked this conversation as resolved.
Outdated
"Please use non-capturing groups (?:...) instead"
)

gid = s.opengroup()
Comment thread
Abhi210 marked this conversation as resolved.
p.append(_parser.SubPattern(s, [
(SUBPATTERN, (gid, 0, 0, _parser.parse(phrase, flags))),
(SUBPATTERN, (gid, 0, 0, sub_pattern)),
]))
s.closegroup(gid, p[-1])
p = _parser.SubPattern(s, [(BRANCH, (None, p))])
Expand Down
19 changes: 19 additions & 0 deletions Lib/test/test_re.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1639,6 +1639,25 @@ def s_int(scanner, token): return int(token)
(['sum', 'op=', 3, 'op*', 'foo', 'op+', 312.5,
'op+', 'bar'], ''))

def test_bug_140797(self):
Copy link
Copy Markdown

@wjssz wjssz Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My negligence, please use test_bug_gh140797 as the name.
If no "gh", It may refer to the previous bug tracker.
Sorry for this.

- # bug 140797: remove capturing groups compilation form re.Scanner
+ # gh140797: capturing groups is not allowed in re.Scanner

Copy link
Copy Markdown
Contributor Author

@Abhi210 Abhi210 Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Thank you again for your time and suggestions!

#bug 140797: remove capturing groups compilation form re.Scanner
Copy link
Copy Markdown

@wjssz wjssz Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a space after # in comments.

- #Capturing group throws an error
+ # Capturing group throws an error

And add a space after , in functions arguments.

- with self.assertRaisesRegex(ValueError,msg):
+ with self.assertRaisesRegex(ValueError, msg):

Then looks good to me.

Copy link
Copy Markdown
Contributor Author

@Abhi210 Abhi210 Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! Thank you again! Resolved


#Presence of Capturing group throws an error
lex = [("(a)b", None)]
with self.assertRaises(ValueError):
Copy link
Copy Markdown

@wjssz wjssz Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may check exception message.

msg = "Can not use capturing groups in re.Scanner"
with self.assertRaisesRegex(ValueError, msg):
    ...

Copy link
Copy Markdown
Contributor Author

@Abhi210 Abhi210 Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the suggestion! I have resolved it now. Need to learn a lot!

Scanner(lex)
Copy link
Copy Markdown

@wjssz wjssz Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This saves a line...
My fault again.

- Scanner(lex)
+ Scanner([("(a)b", None)])

Copy link
Copy Markdown
Contributor Author

@Abhi210 Abhi210 Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Testing sure takes the time 😂


#Presence of non-capturing groups should pass normally
s = Scanner([("(?:a)b", lambda scanner, token: token)])
result, rem = s.scan("ab")
self.assertEqual(result,['ab'])
self.assertEqual(rem,'')
Comment thread
serhiy-storchaka marked this conversation as resolved.

#Testing a very complex capturing group
Comment thread
Abhi210 marked this conversation as resolved.
Outdated
pattern= "(?P<name>a)"
with self.assertRaises(ValueError):
Scanner([(pattern, None)])

def test_bug_448951(self):
# bug 448951 (similar to 429357, but with single char match)
# (Also test greedy matches.)
Expand Down
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/Library/2025-11-03-16-23-54.gh-issue-140797.DuFEeR.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
The re.Scanner class now forbids regular expressions containing capturing
Copy link
Copy Markdown
Member

@serhiy-storchaka serhiy-storchaka Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mention that that class is undocumented. You can also use some formatting, even if the link does not work: :class:`!re.Scanner`.

Copy link
Copy Markdown
Contributor Author

@Abhi210 Abhi210 Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Thank you 😊

groups in its lexicon patterns. Patterns using capturing groups could
previously lead to crashes with segmentation fault. Use non-capturing groups
(?:...) instead.
Loading