-
-
Notifications
You must be signed in to change notification settings - Fork 32.1k
gh-124130: Fix a bug in matching regular expression \B in empty string #127007
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-124130: Fix a bug in matching regular expression \B in empty string #127007
Conversation
RE implementations in other programming languages such as Perl. | ||
This behavior is kept for compatibility reasons. | ||
.. versionchanged:: 3.14 | ||
``\B`` now matches the whole empty string. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fix is LGTM.
Some users may not be sure what "the whole empty string" is.
I suggest explaining in more detail, and no need to use "whole" for "empty string".
\B
used to be unable to match empty string. Now it can match, this behavior is consistent with mainstream languages.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know why you use "whole", the \B
description:
\B Matches the empty string, but only when it ...
You want to distinguish from "empty string" here.
Let me think about how to describe it more clearly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is still difficult to avoid contradiction with "\B Matches the empty string, but only when it ...".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, "the empty string" means two different things there. The docs you're removing don't do a very good job either. Should be disambiguated somehow, for example
\B now matches if the input string is empty.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about this:
\B used to be unable to match 0-length string ""
/b""
. Now it can match, this behavior is consistent with mainstream languages.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
\B now matches if the input string is empty.
@Alcaro uses the word "input", it looks fine. Candidate:
\B
now matches empty input string ""
/b""
, this behavior is consistent with RE implementations in mainstream programming languages.
The previous test: #124130 (comment)
After that, I tested ASCII mode for "a", Unicode mode for "ю".
The results are same except in Perl v5.40.0.
Tested with: regex module, openjdk, .net, rust, ruby, php, pcre2.
javascript/golang only support ASCII word (for \w\W\b\B), even in Unicode mode.
It seems Perl v5.40.0 has bug in (\B
+ Unicode_mode).
Given this, it's not recommended to mention Perl in doc.
Save it to file perl.pl
, and run: perl perl.pl
Click to see Perl script
print "Perl version: $]\n\n";
# test \b in ASCII mode
my $n = () = ( "" =~ /\b/g );
print "\\b \"\" matches: $n\n";
my $n = () = ( "a" =~ /\b/g );
print "\\b \"a\" matches: $n\n";
my $n = () = ( "=" =~ /\b/g );
print "\\b \"=\" matches: $n\n";
print "~~~~~~~~~~~~\n";
# test \B in ASCII mode
my $n = () = ( "" =~ /\B/g );
print "\\B \"\" matches: $n\n";
my $n = () = ( "a" =~ /\B/g );
print "\\B \"a\" matches: $n\n";
my $n = () = ( "=" =~ /\B/g );
print "\\B \"=\" matches: $n\n";
print "\nxxxx ASCII mode above / Unicode mode below xxxx\n\n";
# test \b in Unicode mode
my $n = () = ( "" =~ /\b/gu );
print "\\b \"\" matches: $n\n";
my $n = () = ( "ю" =~ /\b/gu );
print "\\b \"ю\" matches: $n\n";
my $n = () = ( "=" =~ /\b/gu );
print "\\b \"=\" matches: $n\n";
print "~~~~~~~~~~~~\n";
# test \B in Unicode mode
my $n = () = ( "" =~ /\B/gu );
print "\\B \"\" matches: $n\n";
my $n = () = ( "ю" =~ /\B/gu );
print "\\B \"ю\" matches: $n <- other REs get 0, it seems a bug.\n";
my $n = () = ( "=" =~ /\B/gu );
print "\\B \"=\" matches: $n\n";
# explanation
print "\n(\\B + Unicode_mode) behaves inconsistently at the begin/end of a string:\n";
print "\n/^\\B/gu NOT IN \"ю\":\n";
my $n = () = ( "ю" =~ /^\B/gu );
print "matches: $n\n";
print "\n/\\B\$/gu IN \"ю\":\n";
my $n = () = ( "ю" =~ /\B$/gu );
print "matches: $n\n";
Script output:
Perl version: 5.040000
\b "" matches: 0
\b "a" matches: 2
\b "=" matches: 0
~~~~~~~~~~~~
\B "" matches: 1
\B "a" matches: 0
\B "=" matches: 2
xxxx ASCII mode above / Unicode mode below xxxx
\b "" matches: 0
\b "ю" matches: 2
\b "=" matches: 0
~~~~~~~~~~~~
\B "" matches: 1
\B "ю" matches: 1 <- other REs get 0, it seems a bug.
\B "=" matches: 2
(\B + Unicode_mode) behaves inconsistently at the begin/end of a string:
/^\B/gu NOT IN "ю":
matches: 0
/\B$/gu IN "ю":
matches: 1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sorry, Perl doesn't have (\B+Unicode_mode) bug.
If add a line use utf8;
at the beginning of Perl script, it works as expected.
ref: https://perldoc.perl.org/utf8
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The overall change LGTM, but I didn't review Modules/_sre/sre_lib.h
(I don't know this code).
Misc/NEWS.d/next/Library/2024-11-19-10-46-57.gh-issue-124130.OZ_vR5.rst
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one has a conflict (but I don't want to fix it for you since I don't know if you have local commits) but once it's solved, LGTM.
📚 Documentation preview 📚: https://cpython-previews--127007.org.readthedocs.build/