Rustgen: ability to generate rust packages with ergonomic python bindings #2916

kervel · 2025-09-22T13:16:59Z

This PR makes it easier to use rustgen to generate rust crates with a python API:

add the pyo3-stub-gen crate to the python3 generator, so that a python stub module is generated for a rust crate (this means the rust API will have python type hints on it, and you can use your IDEs code completion and type checker)
add an extra flag to rustgen to make the rustgen more friendly for iterative development (now i can split the package in an autogenerated part and a handwritten part, and regenerate without being afraid my handwritten content will be lost)
add a demo function to load yaml using rust in the lib.rs handwritten part, that can be further tweaked (or removed if not needed) by the user
document the whole flow of creating a rust crate with python API.

codecov · 2025-09-29T08:17:34Z

Codecov Report

❌ Patch coverage is 80.70175% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.85%. Comparing base (ec19f23) to head (cc59180).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
linkml/generators/rustgen/rustgen.py	75.64%	8 Missing and 11 partials ⚠️
linkml/generators/rustgen/template.py	93.75%	0 Missing and 2 partials ⚠️
linkml/generators/rustgen/cli.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2916      +/-   ##
==========================================
+ Coverage   80.43%   83.85%   +3.42%     
==========================================
  Files         141      141              
  Lines       15715    15807      +92     
  Branches     3187     3205      +18     
==========================================
+ Hits        12640    13255     +615     
+ Misses       2408     1838     -570     
- Partials      667      714      +47

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

linkml/generators/rustgen/rustgen.py

Add --handwritten-lib flag that redirects generated sources into src/generated, leaves a reusable lib.rs shim, and exposes a YAML loading helper to Python. Update templates, stubgen binary, and rustgen tests accordingly, and document the end-to-end workflow including stub generation before building wheels.

tfliss · 2025-10-08T01:15:37Z

Ok, I read through all the code today, with an AI assist to explain some of the rust / crate bits. rustgen.py feels like it's getting a little long and I left a few other comments, probably not anything I'll hold up the PR on. Tomorrow I'll try running the tests and a cli example and maybe try it out in a colab notebook. For the most part this looks good and very useful. Thank you.

kervel · 2025-10-08T09:59:55Z

hi @tfliss .. is it possible you forgot to submit your PR review ? i don't see the comments ..

linkml/generators/rustgen/templates/enum.rs.jinja

tfliss · 2025-10-07T23:44:35Z

linkml/generators/rustgen/templates/lib_shim.rs.jinja

+
+{% if root_struct_name %}
+#[cfg(feature = "serde")]
+/// Example helper that loads a YAML document into the root class. Edit or extend as needed.


Is a 'yaml' feature needed here specifically in addition to regular serde? Maybe this being an example it isn't that important.

well the "serde" feature flag actually brings in a serde_yml dependency:

SERDE_IMPORTS = Imports( imports=[ Import( module="serde", version="1.0", features=["derive"], objects=[ ObjectImport(name="Serialize"), ObjectImport(name="Deserialize"), ObjectImport(name="de::IntoDeserializer"), ], feature_flag="serde", ), Import(module="serde-value", version="0.7.0", objects=[ObjectImport(name="Value")]), Import(module="serde_yml", version="0.0.12", feature_flag="serde", alias="_"), Import(module="serde_path_to_error", version="0.1.17", objects=[], feature_flag="serde"), ] )

so i think that's fine..

linkml/generators/rustgen/templates/lib_shim.rs.jinja

tfliss · 2025-10-07T23:57:50Z

linkml/generators/rustgen/templates/struct_or_subtype_enum.rs.jinja

 {% endif %}
+
+#[cfg(feature = "stubgen")]
+::pyo3_stub_gen::impl_stub_type!({{ enum_name }} = {{ struct_names | join(' | ') }});


This might be an edge case, or even not allowed in linkml, but if the enum is empty the join will probably generate incorrect syntax. Maybe caught earlier? I saw it in a couple of places, and maybe also the case for some for loops. Won't hold up the PR on this, though. I'm curious if there's a compliance case for empty enums.

tis is about the OrSubtype enums not about regular linkml enums ... these enums are only generated when a class has at least one subtype.

It is needed to model slot range where a slot value can be something or a subtype of that something. rust has no struct inheritance ...

i'll treat the other comments by changing / clarifying code .

tfliss · 2025-10-08T00:04:17Z

linkml/generators/rustgen/templates/stub_gen.rs.jinja

+    let mut issues = Vec::new();
+
+    for (name, module) in &stub.modules {
+        let normalized_name = name.replace('-', "_");


Is this name-cleaning going to be robust enough? Consider at least pulling the normalize into a helper function that is a clear place for future improvements.

Should also check for i.e. * in names that are converted to paths.

ok the helper is there now. its in the rust code not the python code, and its a bit more involved than i originally tought. but its there.

tfliss · 2025-10-08T01:08:06Z

linkml/generators/rustgen/rustgen.py

+                    bf.write(rendered_bin)
+
+        if self.handwritten_lib:
+            shim_path = src_dir / "lib.rs"


Can the path elements be checked for i.e. '..' and '/' and the construction of the path, mkdir, etc. be generalized somewhat? It seems to be repeated. I actually need to look at this in panderagen soon, so wonder if that could go into a shared module.

i have done something, not sure if it is what you wanted.

tfliss · 2025-10-09T00:53:25Z

docs/generators/rust.rst

+
+      import personinfo
+      # Update the path below to point at your data file.
+      container = personinfo.load_yaml_container("path/to/data.yaml")


can you point this to say one of the tutorial person files or some other specific file? I got None when I used examples/tutorial/tutorial07/personinfo.yaml

I probably could find or make one to use, but a specific file would be easier for new users.

Interesting, examples/PersonSchema/data/example_personinfo_data.yaml loads the file of course, but prints [<builtins.Person object at 0x10cabd110>, <builtins.Person object at 0x10cabd2c0>]
I guess this is ok for now, seems odd that Person is defined in builtins.

Ok, I just changed my version of this script fragment to the following and to me it feels better for a quick-start example. Consider using something like this:

container = personinfo.load_yaml_container("examples/PersonSchema/data/example_personinfo_data.yaml") print(container.persons[0].name)

I ran the generator and coverage and it all works fine. I'm marking my review 'changes requested' because you said you're making minor documentation / comment edits and code clarification. Please request a new review when that's done and I'll approve.

Interesting, examples/PersonSchema/data/example_personinfo_data.yaml loads the file of course, but prints [<builtins.Person object at 0x10cabd110>, <builtins.Person object at 0x10cabd2c0>]

i just found out that you can tweak the module name (so that its no longer builtins.) for generated classes. however, its a bit involved (change every #pyclass if i see) and then i have hardcoded module names all over the place. so i'd rater leave it as is for now

the "fix" would be to write:

#[pyclass(module = "mymodule")]

instead of just pyclass.. but then for instance renaming the module becomes next to impossible (having to manually edit generated code). so i'd argue that the fix is worse than the problem.

tfliss

Ok, marking 'request changes' since you said you're making minor edits based on review comments. Overall it looks great and the functionality is useful. Thank you.

tfliss · 2025-10-09T16:50:09Z

docs/generators/rust.rst

+
+      import personinfo
+      # Update the path below to point at your data file.
+      container = personinfo.load_yaml_container("path/to/data.yaml")


Ok, I just changed my version of this script fragment to the following and to me it feels better for a quick-start example. Consider using something like this:

container = personinfo.load_yaml_container("examples/PersonSchema/data/example_personinfo_data.yaml") print(container.persons[0].name)

I ran the generator and coverage and it all works fine. I'm marking my review 'changes requested' because you said you're making minor documentation / comment edits and code clarification. Please request a new review when that's done and I'll approve.

docs/generators/rust.rst

amc-corey-cox · 2025-10-10T16:06:21Z

@kervel and @tfliss are these comments completed? Is this ready for me to review?

kervel · 2025-10-10T16:13:38Z

Not yet! Will try next week...

kervel · 2025-10-14T08:28:33Z

hello @tfliss , i think i implemented your feedback.

tfliss

@kervel I really appreciate your thoughtful work to implement the suggestions. This looks really great and I will look at re-using your file system methods.

@amc-corey-cox If you or another core dev are able to merge I reviewed this PR extensively, used the python API successfullly, and Frank went above and beyond implementing the comment suggestions.

kervel force-pushed the rustgen-pyo3-stubgen branch 2 times, most recently from 94a0b24 to 5a24a92 Compare September 22, 2025 13:51

kervel added the generator-rust label Sep 25, 2025

kervel force-pushed the rustgen-pyo3-stubgen branch from 5a24a92 to 993c252 Compare September 29, 2025 08:14

kervel changed the title ~~[rustgen] generate stubs for python type hints~~ Rustgen: ability to easily generate rust packages with ergonomic python bindings Sep 29, 2025

kervel force-pushed the rustgen-pyo3-stubgen branch from 993c252 to 1e4504a Compare September 29, 2025 09:46

kervel marked this pull request as ready for review September 29, 2025 09:50

kervel requested a review from tfliss September 29, 2025 09:50

kervel force-pushed the rustgen-pyo3-stubgen branch 2 times, most recently from 94da5cc to 9b15e71 Compare September 29, 2025 10:55

kervel changed the title ~~Rustgen: ability to easily generate rust packages with ergonomic python bindings~~ Rustgen: ability to generate rust packages with ergonomic python bindings Sep 29, 2025

kervel commented Sep 29, 2025

View reviewed changes

linkml/generators/rustgen/rustgen.py Show resolved Hide resolved

kervel force-pushed the rustgen-pyo3-stubgen branch from 9b15e71 to 355ec22 Compare September 29, 2025 11:14

kervel mentioned this pull request Sep 30, 2025

rustgen: minor cleanups #2919

Merged

kervel force-pushed the rustgen-pyo3-stubgen branch from 355ec22 to 5bfcbc3 Compare October 3, 2025 10:27

kervel mentioned this pull request Oct 7, 2025

fix compile errors in rust crate generated from tfliss toy schema #2930

Merged

tfliss reviewed Oct 8, 2025

View reviewed changes

tfliss reviewed Oct 9, 2025

View reviewed changes

tfliss requested changes Oct 9, 2025

View reviewed changes

Implement improvements suggested by @tfliss

e0e99d9

kervel requested a review from tfliss October 14, 2025 08:28

tfliss approved these changes Oct 14, 2025

View reviewed changes

Merge branch 'main' into rustgen-pyo3-stubgen

c929bc0

Merge branch 'main' into rustgen-pyo3-stubgen

cc59180

kevinschaper merged commit 551d2da into main Oct 16, 2025
23 checks passed

kevinschaper deleted the rustgen-pyo3-stubgen branch October 16, 2025 17:19

amc-corey-cox restored the rustgen-pyo3-stubgen branch October 16, 2025 21:03

amc-corey-cox deleted the rustgen-pyo3-stubgen branch October 16, 2025 21:03

Rustgen: ability to generate rust packages with ergonomic python bindings #2916

Rustgen: ability to generate rust packages with ergonomic python bindings #2916

Uh oh!

Conversation

kervel commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

tfliss commented Oct 8, 2025

Uh oh!

kervel commented Oct 8, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tfliss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amc-corey-cox commented Oct 10, 2025

Uh oh!

kervel commented Oct 10, 2025

Uh oh!

kervel commented Oct 14, 2025

Uh oh!

tfliss left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kervel commented Sep 22, 2025 •

edited

Loading

codecov bot commented Sep 29, 2025 •

edited

Loading