feat: enable local json string validations #1614
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8f9e7bc to
7dc54ae
Compare
3de56f1 to
6643f93
Compare
fbe0d40 to
f668cce
Compare
bigframes/core/local_data.py
Outdated
| @@ -150,6 +151,38 @@ def validate(self): | |||
| f"Field {bf_field} has arrow array type: {arrow_type}, expected type: {expected_arrow_type}" | |||
| ) | |||
|
|
|||
| if include_context: | |||
There was a problem hiding this comment.
content, not context
There was a problem hiding this comment.
Fixed. Thanks for catching that.
| values = array.to_pandas() | ||
| for data in values: | ||
| # Skip scalar null values to avoid `TypeError` from json.load. | ||
| if not utils.is_list_like(data) and pd.isna(data): |
There was a problem hiding this comment.
Isn't everything a string or a null? how do you get a list-like?
There was a problem hiding this comment.
Now, the data is formatted using pyarrow storage, so these checks are no longer necessary. They are only required to handle values like None, or [] in pandas.
There was a problem hiding this comment.
Re-adding null value checks to ensure compatibility with Python 3.9 tests.
bigframes/core/local_data.py
Outdated
| @@ -329,3 +362,15 @@ def _physical_type_replacements(dtype: pa.DataType) -> pa.DataType: | |||
| if dtype in _ARROW_MANAGED_STORAGE_OVERRIDES: | |||
| return _ARROW_MANAGED_STORAGE_OVERRIDES[dtype] | |||
| return dtype | |||
|
|
|||
|
|
|||
| def _is_valid_json_series(s: pd.Series): | |||
3cea4f7 to
58e672a
Compare
TrevorBergeron
approved these changes
Apr 17, 2025
58e672a to
772e490
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes internal issue 401055880 🦕