py/py.mk: Enable makefile for USER_C_MODULES path. #16960

Carglglz · 2025-03-19T02:48:34Z

This enables setting a file path to select user C modules in a USER_C_MODULES directory the same way CMake works.

The file is a .mk file with the user modules required.

USER_C_MODULES=<path/to/file>/my_user_modules.mk

in my_user_modules.mk:

MODULES_PATH := ../../examples/usercmodule
MODULES := cexample
MODULES += cppexample
MODULES += subpackage

$(info USER_C_MODULES found in $(MODULES_PATH): $(MODULES))
$(foreach module, $(MODULES), \
    $(eval USERMOD_DIR = $(patsubst %/,%,$(MODULES_PATH))/$(module))\
    $(info Including User C Module from $(USERMOD_DIR))\
	$(eval include $(USERMOD_DIR)/micropython.mk)\
)

Note that this doesn't change current behaviour i.e. providing a directory path works as expected.

Summary

Testing

Trade-offs and Alternatives

codecov · 2025-03-19T02:54:56Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.57%. Comparing base (fa393fe) to head (64e7618).
Report is 34 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #16960      +/-   ##
==========================================
+ Coverage   98.54%   98.57%   +0.02%     
==========================================
  Files         169      169              
  Lines       21946    21968      +22     
==========================================
+ Hits        21626    21654      +28     
+ Misses        320      314       -6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-03-19T02:56:19Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

stinos · 2025-03-19T07:55:55Z

The file is a simple text file with the user modules required, e.g.

Can multiple text files be supplied?

Note that this doesn't change current behaviour.

I think it does: originally there'de be an error message if passing a non-existing path?

Anyway, isn't it somewhat confusing that USER_C_MODULES can either be the modules or a .txt, and that implementing it requires 2 different code paths? How about allowing both? E.g. if there would be a USER_C_MODULES_DEF or something like that, to point to a text file, you could in the .mk probably build the USER_C_MODULES variable by appending each value from USER_C_MODULES_DEF to it and leave the rest of the code the same instead of having to loop over MODULE_PATHS in a slightly different way.

andrewleech · 2025-03-19T08:47:29Z

Taking a step back, I gather the goal here is to provide a mechanism for specifying / listing multiple C modules to include in a build that don't necessarily need to be in flat folders side by side like is currently required.

I've wanted something like this myself, particularly to support different folder depths / locations so some can be in arbitrarily structured submodules.

For some reason specifying just a list of paths in a text file like this "feels" wrong to me, though I'm not sure why, I'm possibly being quite unfair.

Basically what I want is the manifest.py system but for the C modules, which leads me to think perhaps it'd be cleaner to extend the existing manifest.py handlers to also support C modules?

The biggest benefit of the manifest system I can think of is that paths are always relative to the manifest file itself, or you have the variables you can use to start from the micropython root dir etc; having something like this would remove a lot of the confusion / discrepancy that currently exists between ports and the "current directory" that relative paths start from.

Gadgetoid · 2025-03-19T09:27:19Z

It feels like the most reasonable (and probably controversial?) answer is: use CMake.

I list multiple C modules frequently across a dozen or more builds just by including them into a CMake file... this feature is already implicitly supported since there's no restriction on what you can put into a micropython.cmake or usercmodules.cmake or my_custom_modules.cmake or whatever you want to name it.

Eg:

https://github.com/pimoroni/pimoroni-pico/blob/main/micropython/modules/micropython-pico.cmake

You can mix and match modules to your heart's content, conditionally enable modules, conditionally configure modules and pretty much anything CMake would ordinarily allow you to do.

Want to include modules based on a variable you've set in a board level configuration file? Sure!

https://github.com/pimoroni/unicorn/blob/54c20aa9d71e4c678f96db1d17ab046a6215edb7/boards/usermod-common.cmake#L38-L39

Want to conditionally include modules based on build target? Go for it!

https://github.com/pimoroni/unicorn/blob/54c20aa9d71e4c678f96db1d17ab046a6215edb7/boards/usermod-common.cmake#L45-L55

Want to enable modules based on the selected board? Go nuts!

https://github.com/pimoroni/pga/blob/98b58dea54636658968ec0f2a9bc67511e3d1527/modules/default.cmake#L21-L23

Reinventing the wheel here is wasted effort, but we should probably document (or provide an example of) this mechanism for supplying multiple modules and maybe it's long past time that py/py.mk based builds were migrated to CMake? Feels like a better application of effort than bolting stuff onto make until it kinda resembles CMake and requires a whole different set of documentation and a whole additional support effort.

Carglglz · 2025-03-20T15:47:03Z

Based on the feedback I've decided to update this and now the file is a makefile so one can add its own logic, the only requirement is to set USER_C_MODULES to the actual path and set MODULE_PATHS.

The file is a simple text file with the user modules required, e.g.
Can multiple text files be supplied?

Note that this doesn't change current behaviour.
I think it does: originally there'de be an error message if passing a non-existing path?

@stinos I guess now you could include multiple makefiles recursively. And no, this still checks first if the path is a directory, if it isn't then checks if the path exists, assuming it is a makefile.

For some reason specifying just a list of paths in a text file like this "feels" wrong to me, though I'm not sure why, I'm possibly being quite unfair.

@andrewleech I agree with this and I've updated the PR accordingly 👍🏼

Basically what I want is the manifest.py system but for the C modules, which leads me to think perhaps it'd be cleaner to extend the existing manifest.py handlers to also support C modules?

The biggest benefit of the manifest system I can think of is that paths are always relative to the manifest file itself, or you have the variables you can use to start from the micropython root dir etc; having something like this would remove a lot of the confusion / discrepancy that currently exists between ports and the "current directory" that relative paths start from.

That would be nice but it would probably take considerable more effort than this tiny change.

@Gadgetoid I think that now with the makefile you could add the same logic right?

Gadgetoid · 2025-03-21T08:00:36Z

@Carglglz this feels like a much more logical approach, thank you!

dpgeorge · 2025-06-16T02:26:01Z

py/py.mk

+$(error USER_C_MODULES doesn't exist: $(abspath $(USER_C_MODULES)))
+else
+# USER_C_MODULES is a .mk file
+$(eval include $(USER_C_MODULES))


Instead of this user .mk file setting MODULE_PATHS, wouldn't it be simpler if that .mk file just did what it needed to do, eg included other .mk files? Then it would work almost the same as the cmake version.

In that case the user .mk file would need to be included below, after setting the SRC_USERMOD variables to empty.

dpgeorge · 2025-06-19T01:18:24Z

@Carglglz thanks for updating. Did you get a chance to test this new version?

Carglglz · 2025-06-19T16:42:49Z

@dpgeorge yes, I've tested

 $ make USER_C_MODULES=../../examples/usercmodule

and

 $ make USER_C_MODULES=../../examples/usercmodule/user_c_modules.mk

they both work, I've also updated the docs (feel free to change anything) and let me know if I need to add this method to unix port coverage somehow? 👀

dpgeorge · 2025-06-24T13:15:35Z

examples/usercmodule/user_c_modules.mk

@@ -0,0 +1,12 @@
+MODULES_PATH := ../../examples/usercmodule


Does this ../../ bit of the path need to be hard coded here? Doesn't it mean that this example must be ../../ relative to where the port is?

If so, that won't work for general C libraries, eg ulab. There needs to be a variable that's available in this file which is the current path, like $(USERMOD_DIR).

Does this ../../ bit of the path need to be hard coded here? Doesn't it mean that this example must be ../../ relative to where the port is?

No, $(USER_C_MODULES_PATH) can be used instead which is

USER_C_MODULES_PATH := $(dir $(USER_C_MODULES))

at this point

dpgeorge · 2025-06-24T13:16:28Z

examples/usercmodule/user_c_modules.mk

+    $(eval USERMOD_DIR = $(patsubst %/,%,$(MODULES_PATH))/$(module))\
+    $(info Including User C Module from $(USERMOD_DIR))\
+	$(eval include $(USERMOD_DIR)/micropython.mk)\
+)


For this example, I'd suggest keeping it as minimal as possible, and similar to the cmake version. Eg just:

include cexample/micropython.mk include cppexample/micropython.mk include subpackage/micropython.mk

I agree with making it similar to the cmake version however then both versions will differ, i.e. this one won't show any information of which modules are included, and also $(USERMOD_DIR) needs to be set for each module.

So you will end with something like this:

$(eval USERMOD_DIR = $(patsubst %/,%,$(USER_C_MODULES_PATH))/cexample)\ $(info Including User C Module from $(USERMOD_DIR))\ $(eval include $(USERMOD_DIR)/micropython.mk)\ $(eval USERMOD_DIR = $(patsubst %/,%,$(USER_C_MODULES_PATH))/cppexample)\ $(info Including User C Module from $(USERMOD_DIR))\ $(eval include $(USERMOD_DIR)/micropython.mk)\ $(eval USERMOD_DIR = $(patsubst %/,%,$(USER_C_MODULES_PATH))/subpackage)\ $(info Including User C Module from $(USERMOD_DIR))\ $(eval include $(USERMOD_DIR)/micropython.mk)\

So I think the single foreach loop is better, but let me know if you prefer this version 👍🏼

This enables setting a file path to select user C modules in a USER_C_MODULES directory, the same way CMake works. The file is a .mk file with the user modules required. USER_C_MODULES=<path/to/file>/user_c_modules.mk See examples/usercmodule/user_c_modules.mk for reference Signed-off-by: Carlos Gil <[email protected]>

Carglglz force-pushed the file_user_c_modules branch from 28f1878 to 4e637e7 Compare March 20, 2025 15:17

Carglglz changed the title ~~py/py.mk: Enable file for USER_C_MODULES build.~~ py/py.mk: Enable makefile for USER_C_MODULES path. Mar 20, 2025

dpgeorge added the py-core Relates to py/ directory in source label Mar 27, 2025

dpgeorge reviewed Jun 16, 2025

View reviewed changes

dpgeorge added this to the release-1.26.0 milestone Jun 16, 2025

Carglglz force-pushed the file_user_c_modules branch 2 times, most recently from 8dc1ae7 to 334752f Compare June 17, 2025 15:07

Carglglz force-pushed the file_user_c_modules branch 2 times, most recently from a11c637 to 328ab76 Compare June 19, 2025 16:38

dpgeorge reviewed Jun 24, 2025

View reviewed changes

Carglglz force-pushed the file_user_c_modules branch from 328ab76 to 64e7618 Compare June 24, 2025 15:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

py/py.mk: Enable makefile for USER_C_MODULES path. #16960

py/py.mk: Enable makefile for USER_C_MODULES path. #16960

Uh oh!

Carglglz commented Mar 19, 2025 •

edited

Loading

Uh oh!

codecov bot commented Mar 19, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

stinos commented Mar 19, 2025

Uh oh!

andrewleech commented Mar 19, 2025

Uh oh!

Gadgetoid commented Mar 19, 2025 •

edited

Loading

Uh oh!

Carglglz commented Mar 20, 2025

Uh oh!

Gadgetoid commented Mar 21, 2025

Uh oh!

dpgeorge Jun 16, 2025

Uh oh!

dpgeorge commented Jun 19, 2025

Uh oh!

Carglglz commented Jun 19, 2025 •

edited

Loading

Uh oh!

dpgeorge Jun 24, 2025

Uh oh!

Carglglz Jun 24, 2025

Uh oh!

dpgeorge Jun 24, 2025

Uh oh!

Carglglz Jun 24, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

py/py.mk: Enable makefile for USER_C_MODULES path. #16960

Are you sure you want to change the base?

py/py.mk: Enable makefile for USER_C_MODULES path. #16960

Uh oh!

Conversation

Carglglz commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Uh oh!

codecov bot commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

stinos commented Mar 19, 2025

Uh oh!

andrewleech commented Mar 19, 2025

Uh oh!

Gadgetoid commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Carglglz commented Mar 20, 2025

Uh oh!

Gadgetoid commented Mar 21, 2025

Uh oh!

dpgeorge Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

dpgeorge commented Jun 19, 2025

Uh oh!

Carglglz commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpgeorge Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

Carglglz Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

dpgeorge Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

Carglglz Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Carglglz commented Mar 19, 2025 •

edited

Loading

codecov bot commented Mar 19, 2025 •

edited

Loading

Gadgetoid commented Mar 19, 2025 •

edited

Loading

Carglglz commented Jun 19, 2025 •

edited

Loading

Carglglz Jun 24, 2025 •

edited

Loading