esp32: Update ESP-IDF to v5.2.2, optimise size #15523
Conversation
9200a66 to
d2a2abc
Compare
|
Rebased to pick up the changes in #15476, the numbers in the main post will be slightly wrong now. |
d2a2abc to
03b13cf
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #15523 +/- ##
=======================================
Coverage 98.43% 98.43%
=======================================
Files 163 163
Lines 21293 21293
=======================================
Hits 20959 20959
Misses 334 334 ☔ View full report in Codecov by Sentry. |
|
Code size report: |
|
I've started a discussion on Discord about more completely dropping support for older ESP-IDF in this PR, as there are a bunch of compatibility checks that we can simplify out: Comments welcome. |
9ecdc15 to
ec63829
Compare
|
Have run unit tests, multi_bluetooth and multi_net (Wi-Fi) tests on ESP32 and ESP32-S3. Found one issue introduced by updating from v5.0.4 to v5.2.2 (mbedTLS-related commit in this PR), plus several other broken tests or bugs that were pre-existing (to be submitted in separate PRs). With all these changes present I now see no test failures, apart from some quite infrequent stress test failures. 🎉 I wasn't able to test on ESP32-C3, both the boards I have are quite old (probably predating the C3's public release) and don't boot reliably on v5.2.2. I see some debug logs about missing eFuse calibration data, so I assume this is an issue with my boards and not MicroPython. Have ordered a newer C3 board, but if someone has a C3 at hand do they mind checking that it boots OK please? @dpgeorge Based on @andrewleech's observation in Discord that some people ran into Wi-Fi issues with v5.2, I suggest we keep all versions supported in this PR but switch CI and nightly builds to v5.2.2. If no significant reports are received in a couple of weeks then I'll submit another PR to drop support for older versions and cleanup the code to suit. What do you think? |
I will check a C3.
Sounds good! |
|
I've just compiled and flashed this to a new C3 board (recently received, been meaning to try it) using official IDF v5.2.2 docker |
|
I just ran multi tests between a C3 and S3 (FLASH_4M), both running 5.2.2 @ ec63829 Ran into a slight hiccup creating the keys ( #15545 ) but then: Ah I see that failed test requires extra files created/copied onto the devices, my bad. Ran into a slight hiccup creating the keys ( #15545 ) but then: Urg, now many more tests fail I'm guessing you need to set the time on the devices somehow.... sorry I tried to help ;-D |
Indeed! Try |
Oh that's easier than I was guessing, I forgot about that command thanks! |
extmod/modtls_mbedtls.c
Outdated
| // | ||
| // Starting with mbedTLS 3.5, DER formatted keys must consume the entire keylen | ||
| // argument (no trailing bytes) or parsing fails. However, PEM formatted keys | ||
| // must be submitted with a terminating NUL byte included in the keylen... |
There was a problem hiding this comment.
This fix is actually being worked on separately in #14385.
|
I tested this PR with the two C3 boards that I have:
And IDF 5.2.2 with this PR does not work on either of them. Looking at the DevKitC in particular, which has a ESP32-C3-WROOM-02 module. With this PR and IDF 5.2.2 the output on the /dev/ttyUSB0 serial when I press the RST button is: When I use this PR but with IDF 5.0.5 I get: These are pretty old boards, but I think they should still work with IDF 5.2.2, considering they work with IDF 5.0.5. Comparing to @andrewleech 's C3 test above:
|
|
How interesting! With DevKitC is that via a UART bridge? Maybe it's the uart stdout that's changed? My board is a cheap ESP32 C3 SuperMini which only exposes the built-in usb, this drops out on rst button and comes up as ACM. |
|
There must be a UART-USB bridge, because the USB stays alive over a reset. The flashing output for this board is: |
Agree. These are the same symptoms mine have, so I guess it's more widespread than just internal testing silicon. Will look into it more. |
Mine don't give any errors about efuse. Do your ESP32C3's work with this PR and IDF 5.0.5? |
|
I've tested the latest version of this PR and it now works on my ESP32-C3-DevKitC-02 board. Running all the tests, one fails: |
See #15583 for a proposed fix for that. @projectgus you may want to cherry-pick that commit into this PR to make testing easier. |
a2de9c7 to
cb451bd
Compare
|
Rebased. My "ESP32-C3 SuperMini" board turned up, using the integrated Serial/JTAG. It seems to work correctly with the serial console config changes in this PR, also. Unit tests pass apart from:
Also:
[*] Except my new SuperMini board randomly resets sometimes mid-test for Bluetooth. Espressif DevKit-C3 does not do this. Pre-existing issue, fix in #15610 . @dpgeorge This is ready apart from #15605 and also the mbedTLS certificate parsing fix. I don't think we should change the default IDF version to V5.2.2 without the mbedTLS fix, as cert parsing will fail. I still have a fix commit for that in this branch (86692a5), but I also made a suggestion to the author of the other fix PR here: #14385 (comment) . Do you have a preference for how to proceed on that one? |
cb451bd to
6286f45
Compare
|
(Rebased again, cleaned up the metrics_esp32.py script and updated the numbers in the PR description.) |
|
The mbedtls and stack-check PRs are now merged, so this is ready for a (final?) rebase and review. |
6286f45 to
973c505
Compare
|
@dpgeorge Thanks! Rebased, re-ran unit tests on ESP32, C3, and S3. I didn't rerun additional tests as I don't think there's a high risk of regressions in Wi-Fi or BT since the last test run. |
This new calibration routine exists for S3 in v5.1.1. It works for all platforms in 5.2.1. Signed-off-by: Andrew Leech <[email protected]>
Disable unnecessary IRAM ISR functionality. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>
Replaces the deprecated ESP32 calibration API with the "line" method instead. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>
Saves code size, MicroPython doesn't appear to rely on any of the missing formatters (64-bit integers, c99-style named arguments). Signed-off-by: Angus Gratton <[email protected]>
Signed-off-by: Angus Gratton <[email protected]>
Keeping older versions, however if the update goes well then these may be dropped in the future. Signed-off-by: Angus Gratton <[email protected]>
The ESP-IDF default on C3 is primary UART0, secondary USB serial/jtag. Previously MicroPython configured the primary as USB Serial/JTAG and manually worked with the UART0 console. However UART0 console stopped working this way in v5.2.2. The big change is that CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG is no longer set, as primary console is UART0. However CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG is set and IDF provides a macro CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG_ENABLED which is set if either primary or secondary esp_console is USB serial/jtag. So need to use that macro instead. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <[email protected]>
973c505 to
0a11832
Compare
|
I tested this on a ESP32-C3-DevKitC-02, running the main test suite, and then the network and bluetooth multitests. Everything passes (except |
|
Now merged! Thanks @projectgus for all your effort testing this. |
Summary
This PR updates the esp32 port's ESP-IDF version to v5.2.2 and changes the version build in CI from v5.0.4 to v5.2.2.
This is in anticipation of adding support for new SoCs ESP32-C6 (see #11869) and ESP32-C2 (see #15440) in the linked PRs.
Propose after merging this and changing the nightly version then we wait a couple of weeks and if no regressions are reported then remove support for older ESP-IDF, including some code cleanup to suit.
Code Size
The usual problem with updating ESP-IDF is an increase in code size. This PR applies some config changes to reduce resource usage, and also adds a utility script to quickly measure the static resource usage of multiple boards & ESP-IDF versions.
Current master branch resource usage:
Note that the ESP32-S3 pools IRAM and DRAM, so the last two columns need to be added together to be meaningful.
Resource usage after applying changes in this PR:
Summary:
Testing
Other TODOs
Trade-offs and Alternatives
The main trade-off decision is keeping vs dropping support for older ESP-IDF, particularly as there is a small impact to available DRAM on v5.2.2. However supporting three different minor versions (5.0, 5.1, 5.2) is a lot of additional overhead for MicroPython, and the difference between 5.1 and 5.2 doesn't seem like it really justifies keeping 5.1 support around. Let me know if I'm wrong!