Thanks to visit codestin.com
Credit goes to github.com

Skip to content

natmod: Add support for static module definitions in native modules #17461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

andrewleech
Copy link
Contributor

Summary

Note: This draft PR is pending manual review before it's ready for consideration.

This PR adds infrastructure to support a cleaner way of defining native modules using static patterns similar to user C modules. The traditional mpy_init approach remains fully supported - this provides an optional migration path for cleaner module definitions.

Key Changes

  1. Parser tool (py/make_natmod_init.py): Extracts MP_REGISTER_MODULE declarations and generates mpy_init functions automatically

  2. Build system (py/dynruntime.mk): Auto-detects static module patterns and invokes parser when needed

  3. Helper macros (py/dynruntime.h):

    • MP_OBJ_QSTR_VALUE: Extract QSTR value from object at runtime
    • MP_DYNRUNTIME_REGISTER_GLOBALS_TABLE: Register globals from table
    • MP_DYNRUNTIME_INIT_STATIC_MODULE: Initialize module from definition
  4. Examples:

    • static_module_hybrid: Demonstrates macro-based approach with zero overhead
    • dualmode: Shows how same C source can compile as either user C module or native module
  5. Documentation: Updated docs/develop/natmod.rst with new patterns

Testing

  • Built and tested all example native modules on Unix port (x64)
  • Verified zero runtime/flash overhead with size analysis
  • Tested dual-mode modules in both configurations

Trade-offs and Alternatives

The macro-based approach for existing natmod examples was attempted but reverted due to QSTR runtime limitations in native modules. The static_module_hybrid example shows the working approach using compile-time macro expansion.

pi-anl added 5 commits June 6, 2025 09:59
This adds documentation for a proposed enhancement to allow native
modules to use the same static module definition pattern as user C
modules, reducing boilerplate and improving consistency.

- natmod-static-modules.md: Overview and analysis of current state
- natmod-implementation-plan.md: Detailed implementation strategy

The proposal includes a parser tool to auto-generate init functions,
build system integration, and maintains backward compatibility.
This commit introduces improvements to reduce boilerplate in native
module definitions while maintaining compatibility:

1. Parser tool (py/make_natmod_init.py) - Generates init functions
   for modules using MP_REGISTER_MODULE pattern (experimental).

2. Build system integration - dynruntime.mk can auto-detect static
   module definitions and extract module names from source.

3. Helper macros in dynruntime.h:
   - MP_OBJ_QSTR_VALUE: Extract QSTR from object
   - MP_DYNRUNTIME_REGISTER_GLOBALS_TABLE: Register module globals
   - MP_DYNRUNTIME_INIT_STATIC_MODULE: Simplified registration

4. Working example (static_module_hybrid) demonstrates a practical
   macro-based approach that reduces boilerplate.

The implementation recognizes native module constraints where QSTRs
are runtime values, not compile-time constants. The hybrid approach
provides a cleaner way to define module attributes while working
within these constraints.
This commit adds infrastructure to support a cleaner way of defining
native modules using static patterns similar to user C modules:

1. Added make_natmod_init.py parser tool to extract MP_REGISTER_MODULE
   declarations and generate mpy_init functions automatically.

2. Enhanced dynruntime.mk to auto-detect static module patterns and
   invoke the parser tool when needed.

3. Added helper macros to dynruntime.h:
   - MP_OBJ_QSTR_VALUE: Extract QSTR value from object at runtime
   - MP_DYNRUNTIME_REGISTER_GLOBALS_TABLE: Register globals from table
   - MP_DYNRUNTIME_INIT_STATIC_MODULE: Initialize module from definition

4. Created static_module_hybrid example demonstrating macro-based
   approach that reduces boilerplate while maintaining zero overhead.

5. Updated documentation in docs/develop/natmod.rst with new patterns
   and helper macros.

6. Added dual-mode module examples showing how the same C source can
   be compiled as either a user C module or native module.

The traditional mpy_init approach remains fully supported. The new
infrastructure is optional and provides a migration path for cleaner
module definitions.

Signed-off-by: Assistant <[email protected]>
Copy link

github-actions bot commented Jun 9, 2025

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Copy link

codecov bot commented Jun 9, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.54%. Comparing base (5f058e9) to head (2d04fba).
Report is 35 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #17461   +/-   ##
=======================================
  Coverage   98.54%   98.54%           
=======================================
  Files         169      169           
  Lines       21943    21943           
=======================================
  Hits        21623    21623           
  Misses        320      320           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dpgeorge dpgeorge added py-core Relates to py/ directory in source examples Relates to examples/ directory in source labels Jun 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
examples Relates to examples/ directory in source py-core Relates to py/ directory in source
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants