expr: fix some multibyte issues #8606
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
de55ab3 to
82fde69
Compare
Should make tests/expr/expr-multibyte pass
82fde69 to
2322bc7
Compare
|
GNU testsuite comparison: |
cakebaker
reviewed
Sep 11, 2025
Comment on lines
+289
to
+311
| let pattern_str = match encoding { | ||
| UEncoding::Utf8 => { | ||
| // In UTF-8 locale, try to parse as UTF-8 | ||
| match String::from_utf8(right_bytes.clone()) { | ||
| Ok(s) => { | ||
| check_posix_regex_errors(&s)?; | ||
| s | ||
| } | ||
| Err(_) => { | ||
| // Invalid UTF-8 pattern in UTF-8 locale - use lossy conversion | ||
| let s = String::from_utf8_lossy(&right_bytes).into_owned(); | ||
| check_posix_regex_errors(&s)?; | ||
| s | ||
| } | ||
| } | ||
| } | ||
| UEncoding::Ascii => { | ||
| // In C locale, validate pattern and use lossy conversion | ||
| let s = String::from_utf8_lossy(&right_bytes).into_owned(); | ||
| check_posix_regex_errors(&s)?; | ||
| s | ||
| } | ||
| }; |
There was a problem hiding this comment.
Unless I'm missing something, I think you don't have to distinguish between UEncoding::Utf8 and UEncoding::Ascii. It would allow you to simplify the block to something like:
Suggested change
| let pattern_str = match encoding { | |
| UEncoding::Utf8 => { | |
| // In UTF-8 locale, try to parse as UTF-8 | |
| match String::from_utf8(right_bytes.clone()) { | |
| Ok(s) => { | |
| check_posix_regex_errors(&s)?; | |
| s | |
| } | |
| Err(_) => { | |
| // Invalid UTF-8 pattern in UTF-8 locale - use lossy conversion | |
| let s = String::from_utf8_lossy(&right_bytes).into_owned(); | |
| check_posix_regex_errors(&s)?; | |
| s | |
| } | |
| } | |
| } | |
| UEncoding::Ascii => { | |
| // In C locale, validate pattern and use lossy conversion | |
| let s = String::from_utf8_lossy(&right_bytes).into_owned(); | |
| check_posix_regex_errors(&s)?; | |
| s | |
| } | |
| }; | |
| let pattern_str = String::from_utf8(right_bytes.clone()) | |
| .unwrap_or_else(|_| String::from_utf8_lossy(&right_bytes).into()); | |
| check_posix_regex_errors(&pattern_str)?; |
cakebaker
reviewed
Sep 11, 2025
| Some(&mut region), | ||
| ); | ||
|
|
||
| if let Some(_pos) = pos { |
There was a problem hiding this comment.
I would use is_some() as you don't care about the value:
Suggested change
| if let Some(_pos) = pos { | |
| if pos.is_some() { |
cakebaker
reviewed
Sep 11, 2025
| Some(&mut region), | ||
| ); | ||
|
|
||
| if let Some(_pos) = pos { |
There was a problem hiding this comment.
Same here:
Suggested change
| if let Some(_pos) = pos { | |
| if pos.is_some() { |
cakebaker
reviewed
Sep 11, 2025
| Some(&mut region), | ||
| ); | ||
|
|
||
| if let Some(_pos) = pos { |
There was a problem hiding this comment.
Same here:
Suggested change
| if let Some(_pos) = pos { | |
| if pos.is_some() { |
cakebaker
reviewed
Sep 11, 2025
| @@ -369,6 +277,257 @@ fn check_posix_regex_errors(pattern: &str) -> ExprResult<()> { | |||
| } | |||
| } | |||
|
|
|||
| /// Evaluate a match expression with locale-aware regex matching | |||
| fn evaluate_match_expression(left_bytes: Vec<u8>, right_bytes: Vec<u8>) -> ExprResult<NumOrStr> { | |||
There was a problem hiding this comment.
I would split this function into multiple functions.
There was a problem hiding this comment.
I think that can be split in a "build_regex" and a "find_match" functions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Should make tests/expr/expr-multibyte pass