Summary

Alternative approach to #16901. Fixes #16870.

Rather than having make submodules run CMake to get the submodules list and then update git submodules, have CMake update submodules itself. This means any other build errors will be visible, rather than being masked (as currently the CMake submodules run has its output captured and parsed.)

This effectively reverts commit 22353e9 from #16581, as an empty submodule list is no longer a symptom of any deeper (masked) issue.

It's also possible to simplify out a lot of the checks for ECHO_SUBMODULES as CMake won't check if those files and directories exist until after the UPDATE_SUBMODULES pass finishes. A workaround is still needed for ESP32 as the ESP-IDF framework tests if paths exist when registering a new component, but the workaround can be contained in the esp32_common.cmake file.

This PR also contains a commit to fail with a helpful error message if the pico-sdk is missing. This was originally necessary because I was trying to run the UPDATE_SUBMODULES pass in the normal build directory (to save on running CMake twice), but that actually doesn't quite work as pico-sdk sets some other invalid cache values if directories are missing. However, the helpful error message is still helpful!

Testing

• Ran some local rp2 and esp32 builds in a clean git clone, verified all successful.
• Ran a build with the older IDF component manager in the IDF V5.2.2 docker image, verified still OK (but does still require the workaround from ec527a1). Command line: podman run --rm -t -v "/home/gus/ry/george/micropython/.git:/gitdir:ro" -t docker.io/espressif/idf:v5.2.2 bash -c "git clone -b $(git branch --show-current) /gitdir micropython && make -j -C micropython/mpy-cross && echo '@@@@@@@@@@ submodules' && make -C microp ython/ports/esp32 submodules V=1 BOARD=ESP32_GENERIC_S3 && echo '@@@@@@@@@@ main build' && make -C micropython/ports/esp32 BOARD=ESP32_GENERIC_S3"

Trade-offs and Alternatives

• The make submodules output now includes a lot of CMake build output for a dummy run which never gets to actually build.
• We could make this even simpler and hard-code the GIT_SUBMODULES value in rp2/Makefile and esp32/Makefile. The upside is a much simpler build stage. There are two downsides I can see:
  • Conceptually, this means each port has to know about implementation details (the submodule path) of all its external MP modules.
  • Practically, can no longer filter the list of submodules based on the board to reduce network traffic and disk usage. On ESP32 the submodule list is current the same for all boards so that doesn't have any current impact. On rp2 the list is much shorter for boards without Wi-Fi or networking, so some additional repos would be fetched unnecessarily. I'm not sure how much more traffic this would actually be, though.