py,tools: Add c_module() manifest function for user C modules #18229

andrewleech · 2025-10-09T06:27:10Z

Summary

This PR adds a c_module() function to the manifest system, allowing projects to specify user C module directories directly in manifest.py files rather than requiring USER_C_MODULES on the command line.

Previously, including user C modules required setting USER_C_MODULES on the make/cmake command line, which made it difficult for projects to have consistent builds without modifying Makefiles or requiring specific build commands. The new c_module() function allows C modules to be specified alongside frozen Python modules in the manifest, providing a single location for all firmware dependencies.

Example usage:

c_module("$(MPY_DIR)/examples/usercmodule/cexample")
c_module("$(BOARD_DIR)/drivers/sensor")

The implementation adds early manifest processing during build configuration (before C compilation) in both Make (py/py.mk) and CMake (py/usermod.cmake) build systems. C module paths are extracted via makemanifest.py --list-c-modules and appended to the existing USER_C_MODULES list, maintaining full backward compatibility.

Key features:

Multiple c_module() calls supported for multiple modules
Same $(VAR) path substitution as other manifest functions
Works with both Make and CMake build systems
Full backward compatibility - USER_C_MODULES continues to work unchanged
Both methods can be used together (modules are combined)
Invalid paths fail the build immediately with clear error messages

Testing

Binary equivalence testing:
Verified that binaries built with c_module() are equivalent to those built with USER_C_MODULES:

RP2 (RPI_PICO_W): Byte-for-byte identical (.text, .rodata, .data, .bss all match via SHA256)
- Testing revealed and fixed a bug in ports/rp2/CMakeLists.txt where FROZEN_MANIFEST overrides were not restoring before usermod.cmake inclusion, causing c_module() calls to be silently ignored
- After fix, all three build methods (c_module() in board manifest, FROZEN_MANIFEST override, USER_C_MODULES) produce identical 876048-byte binaries
STM32 (PYBV11): Byte-for-byte identical (all sections match via SHA256)
Unix (coverage): Functionally identical (text/data/bss sizes match, 192-byte rodata difference is gcov path metadata only, verified with objdump/objcopy analysis)

Build system testing:

Unix coverage variant: Make build with cexample, cppexample modules
- c_module() in variant manifest
RP2 RPI_PICO_W: CMake build with cexample, cppexample modules
- manifest_test.py with FROZEN_MANIFEST override
- Tests both c_module() extraction and FROZEN_MANIFEST code path
ESP32 GENERIC: CMake build via mpbuild Docker with cexample, cppexample modules
- manifest_test.py with FROZEN_MANIFEST override
STM32 PYBV11: Make build with cexample, cppexample, subpackage modules
- manifest_test.py with FROZEN_MANIFEST override for c_module() testing
- Existing CI build maintains USER_C_MODULES for backward compatibility testing

Error handling:

Invalid paths: Build fails immediately with error message to stderr
Missing manifest: Handled gracefully (no C modules included)
Combined usage: USER_C_MODULES + c_module() works correctly

CI coverage:

Unix port: c_module() in coverage variant manifest
ESP32 port: c_module() via manifest_test.py (removed USER_C_MODULES)
RP2 port: c_module() via manifest_test.py with FROZEN_MANIFEST override (removed USER_C_MODULES)
STM32 port: Added new ci_stm32_build_cmod job testing c_module() via manifest_test.py
STM32 port: Existing ci_stm32_pyb_build maintains USER_C_MODULES for backward compatibility
Total: 5 CI configurations test C modules (4 with c_module(), 1 with USER_C_MODULES)

Trade-offs and Alternatives

No negative trade-offs identified:

Code size impact: None - feature is build-time only, no runtime code added
Build time impact: Minimal overhead for parsing manifest twice
Backward compatibility: 100% maintained - existing builds unchanged

Implementation notes:

All manifest processing (variables, C module extraction, frozen content generation) consolidated into py/manifest.mk
Both py/py.mk (early C module extraction) and py/mkrules.mk (frozen content generation) include the same file
Guard prevents duplicate processing when included multiple times
Error messages from makemanifest.py are shown directly to user (not hidden in temp files)

Alternative considered: Extending USER_C_MODULES to support manifest file input was rejected as it would conflate two different specification methods and complicate the implementation.

Behavioral difference: USER_C_MODULES scans a directory and includes all modules with micropython.mk/micropython.cmake files, while c_module() requires explicit listing of each module directory. This is intentional - c_module() provides explicit control over which modules are included, preventing unintended module inclusion from directory scans.

github-actions · 2025-10-09T06:37:32Z

Code size report:

  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Gadgetoid · 2025-10-09T08:43:07Z

This is an aggressively sensible change, and as much as I'm bought into the spaghetti of CMake tooling I've built to handle C modules... doing it this way makes so, so much more sense. 👍

codecov · 2025-10-11T11:55:10Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (8995a29) to head (adb0ccf).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #18229       +/-   ##
===========================================
- Coverage   98.39%        0   -98.40%     
===========================================
  Files         171        0      -171     
  Lines       22290        0    -22290     
===========================================
- Hits        21933        0    -21933     
+ Misses        357        0      -357

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Add c_module() function to manifest files to specify user C modules directly in manifests instead of via USER_C_MODULES make/cmake variable. This provides better integration with the manifest system and allows C modules to be specified alongside frozen Python modules in a single configuration file. The c_module() function accepts a path to a directory containing a micropython.cmake file that defines the C module. Example usage in manifest.py: c_module("$(MPY_DIR)/examples/usercmodule/cexample") Signed-off-by: Andrew Leech <[email protected]>

Update py.mk and py.cmake to extract c_module() paths from manifests during the build process. When MICROPY_FROZEN_MANIFEST is set, the build system now runs makemanifest.py with --list-c-modules to extract C module paths from the manifest, then includes them in the build alongside any modules specified via USER_C_MODULES. Update usermod.cmake to process manifest-specified C modules through the same code path as USER_C_MODULES, ensuring consistent handling. Signed-off-by: Andrew Leech <[email protected]>

Add documentation for the new c_module() function that allows specifying user C modules directly in manifest files. Includes usage examples and explanation of how it integrates with the manifest system. Signed-off-by: Andrew Leech <[email protected]>

Replace USER_C_MODULES makefile variable usage with c_module() calls in the coverage variant manifest. This demonstrates the new manifest approach and ensures the coverage build tests c_module() functionality. The build produces identical binaries to the USER_C_MODULES approach. Signed-off-by: Andrew Leech <[email protected]>

Replace USER_C_MODULES usage with c_module() calls in the ESP32 manifest_test.py. This tests the c_module() functionality on ESP32 during CI builds. Signed-off-by: Andrew Leech <[email protected]>

Fix CMakeLists.txt to restore MICROPY_USER_FROZEN_MANIFEST before including usermod.cmake. Previously, when FROZEN_MANIFEST was specified on the command line, it was saved but then overwritten by the board's mpconfigboard.cmake. The restoration didn't occur until after usermod.cmake was included, causing c_module() calls in manifest overrides to be silently ignored. Add manifest_test.py for CI testing that replicates RPI_PICO_W configuration plus c_module() calls. With the CMakeLists.txt fix, this produces identical binaries to having c_module() in the board manifest. Signed-off-by: Andrew Leech <[email protected]>

Add manifest_test.py with c_module() calls for testing during CI builds. This verifies c_module() functionality works correctly on the STM32 port. Signed-off-by: Andrew Leech <[email protected]>

Update CI scripts to test c_module() functionality: - Remove USER_C_MODULES from ESP32 and RP2 CI builds (now using c_module() in manifest_test.py) - Add ci_stm32_build_cmod function to test c_module() on STM32 - Update RP2 CI to use FROZEN_MANIFEST with manifest_test.py This ensures c_module() is tested across multiple ports during CI. Signed-off-by: Andrew Leech <[email protected]>

andrewleech mentioned this pull request Oct 9, 2025

py/py.mk: Enable makefile for USER_C_MODULES path. #16960

Open

dpgeorge added the py-core Relates to py/ directory in source label Oct 11, 2025

pi-anl added 8 commits November 13, 2025 12:32

ports/esp32: Update manifest_test to use c_module().

d19bf73

Replace USER_C_MODULES usage with c_module() calls in the ESP32 manifest_test.py. This tests the c_module() functionality on ESP32 during CI builds. Signed-off-by: Andrew Leech <[email protected]>

ports/stm32: Add manifest_test for c_module() CI testing.

1cd78b7

Add manifest_test.py with c_module() calls for testing during CI builds. This verifies c_module() functionality works correctly on the STM32 port. Signed-off-by: Andrew Leech <[email protected]>

andrewleech force-pushed the manifest_c_module branch from adb0ccf to d90efde Compare November 13, 2025 01:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

py,tools: Add c_module() manifest function for user C modules #18229

py,tools: Add c_module() manifest function for user C modules #18229

andrewleech commented Oct 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 9, 2025

Uh oh!

Gadgetoid commented Oct 9, 2025

Uh oh!

codecov bot commented Oct 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

py,tools: Add c_module() manifest function for user C modules #18229

Are you sure you want to change the base?

py,tools: Add c_module() manifest function for user C modules #18229

Conversation

andrewleech commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Uh oh!

github-actions bot commented Oct 9, 2025

Uh oh!

Gadgetoid commented Oct 9, 2025

Uh oh!

codecov bot commented Oct 11, 2025

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andrewleech commented Oct 9, 2025 •

edited

Loading