Summary

Fixes #16870.

The existing method of grepping the output of CMake still masks errors that can occur if something else in the build isn't working. This is hard to debug. Commit 22353e9 in #16581 added an error message, but the root cause of the failure is still hidden.

Instead of capturing stdout, have CMake write a shell snippet in the build directory and then source it to set GIT_SUBMODULES in Make. This way stdout/stderr are no longer masked off, and the build will fail normally if something goes wrong.

This reverts both commits from #16581:

• Commit 22353e9 (empty submodule list is no longer a symptom of something else)
• Commit ec527a1 (CMake can now return normally instead of fail, so the problem with IDF Component Manager is no longer triggered.)

This work was funded through GitHub Sponsors.

Testing

• Ran rp2 and esp32 builds locally, checked the submodule list was correct in all cases.
• Checked that if CMake fails with an error, the error now prints on the console and the build fails at this step.
• Ran ESP32 container build to check that the issue with older IDF Component Manager hasn't returned. (Chonky command line: podman run --rm -t -v "/home/gus/ry/george/micropython/.git:/gitdir:ro" -t docker.io/espressif/idf:v5.2.2 bash -c "export BOARD=ESP32_GENERIC_S3; git clone -b $(git branch --show-current) /gitdir micropython && make -j -C micropython/mpy-cross && echo '@@@@@@@@@@ submodules' && make -C micropython/ports/esp32 submodules V=1 BOARD=$BOARD && echo '@@@@@@@@@@ main build' && make -C micropython/ports/esp32")

Trade-offs and Alternatives

• The make submodules run now prints some CMake informational messages, so it's noiser. Could potentially add -q to reduce this (or to reduce it unless V=1 is set), but I haven't done this as I don't want to mask useful build context.
• Rather than going back up the stack so Make can run git submodules, we could delegate running git submodules to CMake. The "submodules" target in py/mkrules.mk is very small, we might end up with less lines of code overall by duplicating it in CMake.
• Rather than doing any of this, we could hard-code the GIT_SUBMODULES value in rp2/Makefile and esp32/Makefile. The upside is a much simpler build stage. There are two downsides I can see:
  • Conceptually, this means each port has to know about implementation details (the submodule path) of all its external MP modules.
  • Practically, can no longer filter the list of submodules based on the board to reduce network traffic and disk usage. On ESP32 the submodule list is current the same for all boards so that doesn't have any current impact. On rp2 the list is much shorter for boards without Wi-Fi or networking, so some additional repos would be fetched unnecessarily. I'm not sure how much more traffic this would actually be, though.