Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-124130: Fix a bug in matching regular expression \B in empty string #127007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jan 2, 2025

Conversation

serhiy-storchaka
Copy link
Member

@serhiy-storchaka serhiy-storchaka commented Nov 19, 2024

RE implementations in other programming languages such as Perl.
This behavior is kept for compatibility reasons.
.. versionchanged:: 3.14
``\B`` now matches the whole empty string.

Copy link

@wjssz wjssz Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is LGTM.

Some users may not be sure what "the whole empty string" is.
I suggest explaining in more detail, and no need to use "whole" for "empty string".

\B used to be unable to match empty string. Now it can match, this behavior is consistent with mainstream languages.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know why you use "whole", the \B description:

\B Matches the empty string, but only when it ...

You want to distinguish from "empty string" here.
Let me think about how to describe it more clearly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is still difficult to avoid contradiction with "\B Matches the empty string, but only when it ...".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, "the empty string" means two different things there. The docs you're removing don't do a very good job either. Should be disambiguated somehow, for example

\B now matches if the input string is empty.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about this:

\B used to be unable to match 0-length string ""/b"". Now it can match, this behavior is consistent with mainstream languages.

Copy link

@wjssz wjssz Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

\B now matches if the input string is empty.

@Alcaro uses the word "input", it looks fine. Candidate:

\B now matches empty input string ""/b"", this behavior is consistent with RE implementations in mainstream programming languages.


The previous test: #124130 (comment)
After that, I tested ASCII mode for "a", Unicode mode for "ю".
The results are same except in Perl v5.40.0.

Tested with: regex module, openjdk, .net, rust, ruby, php, pcre2.
javascript/golang only support ASCII word (for \w\W\b\B), even in Unicode mode.


It seems Perl v5.40.0 has bug in (\B + Unicode_mode).
Given this, it's not recommended to mention Perl in doc.

Save it to file perl.pl, and run: perl perl.pl

Click to see Perl script
print "Perl version: $]\n\n";

# test \b in ASCII mode
my $n = () = ( "" =~ /\b/g );
print "\\b \"\"  matches: $n\n";

my $n = () = ( "a" =~ /\b/g );
print "\\b \"a\" matches: $n\n";

my $n = () = ( "=" =~ /\b/g );
print "\\b \"=\" matches: $n\n";

print "~~~~~~~~~~~~\n";

# test \B in ASCII mode
my $n = () = ( "" =~ /\B/g );
print "\\B \"\"  matches: $n\n";

my $n = () = ( "a" =~ /\B/g );
print "\\B \"a\" matches: $n\n";

my $n = () = ( "=" =~ /\B/g );
print "\\B \"=\" matches: $n\n";

print "\nxxxx  ASCII mode above / Unicode mode below xxxx\n\n";

# test \b in Unicode mode
my $n = () = ( "" =~ /\b/gu );
print "\\b \"\"   matches: $n\n";

my $n = () = ( "ю" =~ /\b/gu );
print "\\b \"ю\" matches: $n\n";

my $n = () = ( "=" =~ /\b/gu );
print "\\b \"=\"  matches: $n\n";

print "~~~~~~~~~~~~\n";

# test \B in Unicode mode
my $n = () = ( "" =~ /\B/gu );
print "\\B \"\"   matches: $n\n";

my $n = () = ( "ю" =~ /\B/gu );
print "\\B \"ю\" matches: $n  <- other REs get 0, it seems a bug.\n";

my $n = () = ( "=" =~ /\B/gu );
print "\\B \"=\"  matches: $n\n";

# explanation
print "\n(\\B + Unicode_mode) behaves inconsistently at the begin/end of a string:\n";

print "\n/^\\B/gu NOT IN \"ю\":\n";
my $n = () = ( "ю" =~ /^\B/gu );
print "matches: $n\n";

print "\n/\\B\$/gu IN \"ю\":\n";
my $n = () = ( "ю" =~ /\B$/gu );
print "matches: $n\n";

Script output:

Perl version: 5.040000

\b ""  matches: 0
\b "a" matches: 2
\b "=" matches: 0
~~~~~~~~~~~~
\B ""  matches: 1
\B "a" matches: 0
\B "=" matches: 2

xxxx  ASCII mode above / Unicode mode below xxxx

\b ""   matches: 0
\b "ю"  matches: 2
\b "="  matches: 0
~~~~~~~~~~~~
\B ""   matches: 1
\B "ю"  matches: 1  <- other REs get 0, it seems a bug.
\B "="  matches: 2

(\B + Unicode_mode) behaves inconsistently at the begin/end of a string:

/^\B/gu NOT IN "ю":
matches: 0

/\B$/gu IN "ю":
matches: 1

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, Perl doesn't have (\B+Unicode_mode) bug.
If add a line use utf8; at the beginning of Perl script, it works as expected.
ref: https://perldoc.perl.org/utf8

Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overall change LGTM, but I didn't review Modules/_sre/sre_lib.h (I don't know this code).

Copy link
Member

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one has a conflict (but I don't want to fix it for you since I don't know if you have local commits) but once it's solved, LGTM.

@serhiy-storchaka serhiy-storchaka enabled auto-merge (squash) January 2, 2025 11:44
@serhiy-storchaka serhiy-storchaka merged commit a3711d1 into python:main Jan 2, 2025
41 checks passed
@serhiy-storchaka serhiy-storchaka deleted the re-non-boundary branch January 2, 2025 12:23
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this pull request Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants