Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.57%. Comparing base (fa393fe) to head (64e7618).
Report is 34 commits behind head on master.

@@            Coverage Diff             @@
##           master   #16960      +/-   ##
==========================================
+ Coverage   98.54%   98.57%   +0.02%     
==========================================
  Files         169      169              
  Lines       21946    21968      +22     
==========================================
+ Hits        21626    21654      +28     
+ Misses        320      314       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

The file is a simple text file with the user modules required, e.g.

Can multiple text files be supplied?

Note that this doesn't change current behaviour.

I think it does: originally there'de be an error message if passing a non-existing path?

Anyway, isn't it somewhat confusing that USER_C_MODULES can either be the modules or a .txt, and that implementing it requires 2 different code paths? How about allowing both? E.g. if there would be a USER_C_MODULES_DEF or something like that, to point to a text file, you could in the .mk probably build the USER_C_MODULES variable by appending each value from USER_C_MODULES_DEF to it and leave the rest of the code the same instead of having to loop over MODULE_PATHS in a slightly different way.

Taking a step back, I gather the goal here is to provide a mechanism for specifying / listing multiple C modules to include in a build that don't necessarily need to be in flat folders side by side like is currently required.

I've wanted something like this myself, particularly to support different folder depths / locations so some can be in arbitrarily structured submodules.

For some reason specifying just a list of paths in a text file like this "feels" wrong to me, though I'm not sure why, I'm possibly being quite unfair.

Basically what I want is the manifest.py system but for the C modules, which leads me to think perhaps it'd be cleaner to extend the existing manifest.py handlers to also support C modules?

The biggest benefit of the manifest system I can think of is that paths are always relative to the manifest file itself, or you have the variables you can use to start from the micropython root dir etc; having something like this would remove a lot of the confusion / discrepancy that currently exists between ports and the "current directory" that relative paths start from.

It feels like the most reasonable (and probably controversial?) answer is: use CMake.

I list multiple C modules frequently across a dozen or more builds just by including them into a CMake file... this feature is already implicitly supported since there's no restriction on what you can put into a micropython.cmake or usercmodules.cmake or my_custom_modules.cmake or whatever you want to name it.

Eg:

https://github.com/pimoroni/pimoroni-pico/blob/main/micropython/modules/micropython-pico.cmake

You can mix and match modules to your heart's content, conditionally enable modules, conditionally configure modules and pretty much anything CMake would ordinarily allow you to do.

Want to include modules based on a variable you've set in a board level configuration file? Sure!

https://github.com/pimoroni/unicorn/blob/54c20aa9d71e4c678f96db1d17ab046a6215edb7/boards/usermod-common.cmake#L38-L39

Want to conditionally include modules based on build target? Go for it!

https://github.com/pimoroni/unicorn/blob/54c20aa9d71e4c678f96db1d17ab046a6215edb7/boards/usermod-common.cmake#L45-L55

Want to enable modules based on the selected board? Go nuts!

https://github.com/pimoroni/pga/blob/98b58dea54636658968ec0f2a9bc67511e3d1527/modules/default.cmake#L21-L23

Reinventing the wheel here is wasted effort, but we should probably document (or provide an example of) this mechanism for supplying multiple modules and maybe it's long past time that py/py.mk based builds were migrated to CMake? Feels like a better application of effort than bolting stuff onto make until it kinda resembles CMake and requires a whole different set of documentation and a whole additional support effort.

Based on the feedback I've decided to update this and now the file is a makefile so one can add its own logic, the only requirement is to set USER_C_MODULES to the actual path and set MODULE_PATHS.

The file is a simple text file with the user modules required, e.g.
Can multiple text files be supplied?

Note that this doesn't change current behaviour.
I think it does: originally there'de be an error message if passing a non-existing path?

@stinos I guess now you could include multiple makefiles recursively. And no, this still checks first if the path is a directory, if it isn't then checks if the path exists, assuming it is a makefile.

For some reason specifying just a list of paths in a text file like this "feels" wrong to me, though I'm not sure why, I'm possibly being quite unfair.

@andrewleech I agree with this and I've updated the PR accordingly 👍🏼

Basically what I want is the manifest.py system but for the C modules, which leads me to think perhaps it'd be cleaner to extend the existing manifest.py handlers to also support C modules?

The biggest benefit of the manifest system I can think of is that paths are always relative to the manifest file itself, or you have the variables you can use to start from the micropython root dir etc; having something like this would remove a lot of the confusion / discrepancy that currently exists between ports and the "current directory" that relative paths start from.

That would be nice but it would probably take considerable more effort than this tiny change.

@Gadgetoid I think that now with the makefile you could add the same logic right?

@Carglglz this feels like a much more logical approach, thank you!

@Carglglz thanks for updating. Did you get a chance to test this new version?

@dpgeorge yes, I've tested

 $ make USER_C_MODULES=../../examples/usercmodule 

and

 $ make USER_C_MODULES=../../examples/usercmodule/user_c_modules.mk 

they both work, I've also updated the docs (feel free to change anything) and let me know if I need to add this method to unix port coverage somehow? 👀