Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add JSONValues container for holding Python values as JSON objects if possible, and as pybind11::object otherwise#455

Merged
rapids-bot[bot] merged 44 commits into
nv-morpheus:branch-24.06from
dagardner-nv:david-devin-json-serializable
Apr 3, 2024
Merged

Add JSONValues container for holding Python values as JSON objects if possible, and as pybind11::object otherwise#455
rapids-bot[bot] merged 44 commits into
nv-morpheus:branch-24.06from
dagardner-nv:david-devin-json-serializable

Conversation

@dagardner-nv

@dagardner-nv dagardner-nv commented Mar 13, 2024

Copy link
Copy Markdown
Contributor

Description

  • Add an optional unserializable_handler_fn callback to mrc::pymrc::cast_from_pyobject which will be invoked for any unsupported Python object. Allowing for serializing unsupported object.
    ex:
    pymrc::unserializable_handler_fn_t handler_fn = [](const py::object& source,
                                                       const std::string& path) {
        return nlohmann::json(py::cast<float>(source));
    };

    // decimal.Decimal is not serializable
    py::object Decimal = py::module_::import("decimal").attr("Decimal");
    py::object o       = Decimal("1.0");
    EXPECT_EQ(pymrc::cast_from_pyobject(o, handler_fn), nlohmann::json(1.0));
  • Add JSONValues container class which is an immutable container for holding Python values as nlohmann::json objects if possible, and as pybind11::object otherwise. The container can be copied and moved, but the underlying nlohmann::json object is immutable.

  • Updates nlohmann_json from 3.9 to 3.11 for patch_inplace method.

  • Incorporates ideas from @drobison00's PR Update PythonObjectCache to support get and pop operations, update utils to allow passing of cached python objects. #417 with three key differences:

    1. Changes to the cast_from_pyobject are opt-in when unserializable_handler_fn is provided, otherwise there is no behavior change to the method.
    2. Unserializable objects are stored in JSONValues rather than a global cache.
    3. Does not rely on parsing the place-holder values.

This PR is related to nv-morpheus/Morpheus#1560

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@dagardner-nv dagardner-nv self-assigned this Mar 13, 2024
@dagardner-nv dagardner-nv requested review from a team as code owners March 13, 2024 19:00
@dagardner-nv dagardner-nv added non-breaking Non-breaking change feature request New feature or request labels Mar 13, 2024
Comment thread python/mrc/_pymrc/tests/test_json_values.cpp
@dagardner-nv dagardner-nv marked this pull request as draft March 14, 2024 21:01
@dagardner-nv dagardner-nv marked this pull request as draft March 26, 2024 20:47
@dagardner-nv dagardner-nv marked this pull request as ready for review March 26, 2024 23:01
Comment thread python/mrc/_pymrc/include/pymrc/utilities/json_values.hpp Outdated
@dagardner-nv dagardner-nv marked this pull request as draft March 27, 2024 23:41
@dagardner-nv dagardner-nv added the skip-ci Optionally Skip CI for this PR label Mar 27, 2024
@dagardner-nv dagardner-nv removed the skip-ci Optionally Skip CI for this PR label Apr 1, 2024
@dagardner-nv dagardner-nv changed the base branch from branch-24.03 to branch-24.06 April 1, 2024 18:10
@dagardner-nv dagardner-nv requested a review from mdemoret-nv April 1, 2024 18:14
@dagardner-nv dagardner-nv marked this pull request as ready for review April 1, 2024 18:14
@dagardner-nv dagardner-nv added the conda-build Enables running the conda-build step on a PR label Apr 1, 2024
@codecov

codecov Bot commented Apr 2, 2024

Copy link
Copy Markdown

Codecov Report

Merging #455 (64f7459) into branch-24.06 (bd7955e) will increase coverage by 0.09%.
The diff coverage is 88.70%.

Additional details and impacted files

Impacted file tree graph

@@               Coverage Diff                @@
##           branch-24.06     #455      +/-   ##
================================================
+ Coverage         73.68%   73.77%   +0.09%     
================================================
  Files               400      402       +2     
  Lines             14275    14395     +120     
  Branches           1113     1129      +16     
================================================
+ Hits              10518    10620     +102     
- Misses             3757     3775      +18     
Flag Coverage Δ
cpp 70.39% <94.01%> (+0.16%) ⬆️
py 41.28% <2.41%> (-0.46%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...mrc/_pymrc/include/pymrc/utilities/json_values.hpp 100.00% <100.00%> (ø)
...rc/_pymrc/include/pymrc/utilities/object_cache.hpp 0.00% <ø> (ø)
python/mrc/_pymrc/src/utils.cpp 87.50% <100.00%> (+1.95%) ⬆️
python/mrc/_pymrc/src/utilities/json_values.cpp 87.61% <87.61%> (ø)

... and 4 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd7955e...64f7459. Read the comment docs.

@mdemoret-nv

Copy link
Copy Markdown
Contributor

/merge

@rapids-bot rapids-bot Bot merged commit f4e6266 into nv-morpheus:branch-24.06 Apr 3, 2024
rapids-bot Bot pushed a commit to nv-morpheus/Morpheus that referenced this pull request Apr 5, 2024
* Contains breaking changes for the C++ API 
* No breaking changes for the Python API
* Updates `nlohmann_json from` 3.9 to 3.11 for `patch_inplace` method.
* Requires nv-morpheus/MRC#455 to be merged first
Closes #1560

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1589
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conda-build Enables running the conda-build step on a PR feature request New feature or request non-breaking Non-breaking change

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants