-
-
Notifications
You must be signed in to change notification settings - Fork 8.2k
RFC: Built-in module extending and removing weak links / umodules #9018
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Some other misc thoughts that don't quite fit into the above:
|
MicroPython is not CPython. I do not like the idea of changing names for long established classes, methods, functions. That would by a big pain for people actually using micropython for their projects and might want to update the firmware e.g. for bug fixes. |
@robert-hh Just to clarify, I'm not proposing changing the names of any classes, methods, or functions. Only modules. In terms of backwards compatibility it would be very easy to continue to provide the existing weak links / aliases for ufoo->foo (except in the reverse direction to the current implementation, such that the module would now be called The main breaking change I'm proposing would be that builtins always take precedence and are never searched on the filesystem. |
Thanks for the really clear, well thought-through write-up @jimmo! +1 to remove the ufoo/weak link model. I've found it to be one of the most confusing topics for beginner/intermediate MicroPython users - and one of the more difficult to explain.
I'd prefer not - but I can imagine some compelling cases that could be convincing.
Yes to finding new homes. Not long would be my preference - make painful changes quickly. More important to me: Making breaking changes like this - and their workarounds - very clear in release notes. |
A very compelling use case for me is that the "u" prefix allows us to use existing Python intellesene tools in IDEs to get correct code completion and type hints for the MicroPython version of a module. Existing tools don't have a way of saying "I want to replace the Python standard library with my own
As long as this is possible, then the proposed changes aren't really breaking then are they (other than the aliases may be disabled by default instead of enabled)? |
Here is my perspective and what I use to guide CircuitPython.
So, for 1, we've dropped the utime prefixes and for 2 we've dropped or moved all non-CPython APIs to other modules like Regarding allowing extending builtins, do you really need it? Why can't the calling code be changed to use a different module name or simply I also have run into u* libraries that may have intended to match their CPython equivalent but then varied. I think the u* naming in libraries is a bit of a scapegoat to be similar instead of a strict subset. I find that variation confusing because it doesn't match one's expectations. Leaving the u* naming behind will lead to more different names for different modules and I think that'd be good. |
I'm 100% in favour of this. Extending builtins is a surprisingly common practice in even cpython (in my experience), and even more so in micropython. As @jimmo mentioned this aids in keeping the built-in modules lean while allowing users to extend as needed for their application. The It's true that the both the current mechanism and the proposed change comes with a ram cost in the duplication of the module table with Similarly, the new system could be implemented in C with the built-in getattr/setattr functions; make
I've run into a number of cases where porting cpython code to micropython where the |
Definitely yes, main reason being that most builtins are not 100% CPython-comaptible: the choice is then to either write a new module and provide functions, or to just do what is most convenient and user-friendly in all possible ways (code is CPython compatible, can just use standard CPython docs, no need to learn about another module and/or function name, ...) |
For the unix-lik ports this would require some new mechanism (preferrably CPython-compatible, site.py perhaps) to have something run before everything else. At least I don't think we have that now. Possible (perf not measured) disadvantage is that in practice for most projects I have this would mean importing like 10 modules, always, even when none needed. Option 1 is a no-go as far as I'm concerned, 3 only fixes the 2 and 4 on the other hand are technically nice but have the issue mentioned above, and are not super convenient to write in general, and it also hurts discoverability somewhat (e.g. now if you want to know what's in |
Thanks for the comments!
@dlech OK interesting. This is something I hadn't considered, nor do I have any experience with. My initial reaction is along the lines "surely there's got to be a better way" and to be honest we're already trying to make "use foo not ufoo" the default guidance, and so complicating that with "but ufoo fixes autocompletion" is unfortunate (and also means that you have to choose between supporting extensions-from-Python or autocompletion). How bad is it that the autocomplete just uses Python's full completion? Is this a limitation of all IDE that they can't have project-aware auto-completion sources?
Yes, the only breaking change would be that we would stop the existing extension mechanism from working (i.e. builtins would now always take precedence). Yes. Others will correct me if I'm wrong but I think our philosophy is much more: "MicroPython code can run in CPython with necessary shims, e.g. micropython.const, etc, and CPython fragments and occasionally whole modules should run unmodified.".
@tannewt @andrewleech Yes, I guess this is exactly the crux of this conversation. This whole goal of extending built-in modules is only really necessary to make an existing file work completely unmodified. In the "import foo_ext as foo" case, foo_ext still needs to do "from foo import *" (duplicate dict cost) or the module
This is a good idea, thanks @andrewleech . I think the whole "getattr + builtinimport hook" could be combined into a utility module too, so os_ext.py could look like: import builtin_extension
# This gives us a __getattr__ forwarding to the existing os
# module (either built-in or a previous extesnion), and hooks
# "import os" for further usage to return this module instead.
builtin_extension.apply("os", __name__)
def foo():
pass Here is an implementation of builtin_extension.py to support this -- https://gist.github.com/jimmo/b918b73abc2f5c107b102d2ddb7a7976.
Yes, exactly. This is along the lines of how the existing
@stinos Does Scott's point above with "import foo_ext as foo" work for you? Or, like Andrew, do you want the CPython-compatible file to run exactly as-is without any modifications to import.
@stinos I'm not quite sure I follow. I had imagined that the top of main.py (or for Unix/Windows, whatever your entry point is) would do this.
You could also imagine that some sort of automatically maintained script that imported all your installed extension packages could be useful. But conceptually I see this more as "I'm explicitly enabling the extra functionality I need for my app".
Do any of the approaches solve this? If we're going to make it possible to extend a module, it needs to be implemented across multiple files.
Which one do you want to see code for? The implementation of mutable builtin-modules for 2? |
Yes I'd really prefer this to stay as it is now, such that we can write code which is CPython compatible with the least friction, i.e. just edit just to give an idea, there are 43 matches for
When prototyping, writing (unit)tests etc I just want to create a new file and start writing 'normal' code, then run it (1). I wouldn't want to have to manually add
No, but it's a minor issue. Also because as long as the extension is written in Python most text editor's 'go to symbol' etc will find it anyway.
Yes, to see how it compares in practice with the approach for 4 |
You end up using modules/classes/methods/args that don't exist in MicroPython and your code fails at run time. Then you have to go find the relevant MicroPython documentation to figure out what is going on. This slows down development and is confusing for inexperienced programmers.
Generally, the way these tools work is that you say I want to use X Python runtime. This is specified by the absolute path to python executable and will use that find all site packages, etc this way. Usually this a virtual environment where you have all of the dependencies for your project installed. But since MicroPython is not fully CPython compatible, you can't just plug in the path to MicroPython here or create a MicroPython virtual environment. |
I concur, currently if you have all the libraries "upip installed" or equivalent then just running files have all the libs on the path ready to import. The
Is there a particular ide / autocomplete package you're using currently? Autocomplete and code navigation is something I use daily myself but have just gotten used to the sub-par behaviour of cpython stubs. This certainly isn't ideal though. A proper micropython autocomplete setup is something I'm keen to get working. I know there are some stubbing tools it there that should do most of the work for us, perhaps arranging the outputs of that into something an ide could recognise as a python env would be possible? Maybe that's something that could be built on top of the new manifest system to include copies/links to libraries installed as well. |
We're using Pylance(based on Pyright) in VSCode and also Jedi.
Microbit forked Pyright to make it work in the browser and with their MicroPython API. You can see it in action at https://python.microbit.org/v/beta. Making this work in general for MicroPython with some sort of manifest system as you have describe would be ideal IMHO. |
Some notes on the performance / memory... By removing the filesystem search from importing a builtin, With option 4 (i.e. hooking It takes 3us to do I'm not sure if this is a big concern... if this was performance critical, then sys.modules[ext_name].__getattr__ = micropython.bind(getattr, mod) Even simpler if you just want to make a particular method fast you can write As far as memory overhead, installing the extension costs 224 bytes of RAM for the import hook etc. |
I've always thought of the u-modules as accelerator modules, only using
I'm not arguing to do this exactly, but it seems like there is some precedent that could provide inspiration here.
It seems like implementation could address this, but having the accelerator modules shining through isn't inherently wrong/weird. For example in Python3,
But you can extend an accelerator module
With this pattern in mind, the end user and documentation will always use The difference from PEP 399 is that MicroPython wouldn't actually always use a pure python wrapper to make this work, but this comes down to implementation. The existing weak links approach is just MicroPython's current way of doing it, but this could perhaps be improved. So I guess my point is that having u-modules in the implementation makes sense and has some CPython precedent. As mentioned above, end-users should be able to import End-users (especially beginners) would never have to see or use the u-modules, but library builders can still use them. |
Thanks @laurensvalk
Yes, this is exactly how It's also relevant to #8968 where one of the options being considered is to freeze ssl.py and provide It's a bit difficult to apply this as a precedent to this issue though because you still can't override
My goal here is that I don't think our umodule/weak link system is actually the best way to provide the functionality that users want/need.
I'd be interested to know more about your specific use case with pybricks. Do you provide pre-built firmware with frozen in extensions? Or do your users add them to the filesystem? Are they writing their own or using micropython-lib ones (or other sources?). Would any of the options outlined in this issue work for you & your users? |
Thanks for your response. That led me to a subtlety that may be worth clarifying --- Reading back through this thread, it seems that not all comments about extending builtins are about the same thing. If I understand the posts in favor of extending builtins correctly, most would like to extend uos, not necessarily extend os. But both variants are implicitly or explicitly discussed in this thread, and I'm not sure the replies here always respond to the same thing. I'll follow up with my personal opinion in the next post. |
So with that in mind, I fully agree with the motivation in @tannewt's post, but with a subtle clarification of the implication --- moving away from the
Agreed. My comments so far are mainly about keeping the internal implementation modules (e.g.
I understand the concerns on RAM and build size, but how important is import time?
Most of this doesn't affect Pybricks directly, at least not right now. I wrote this from a generic point of view, and with the backdrop of having written several coding books for kids. Several posts have mentioned that some solutions are easy to explain. I believe it's even better when it's easy to understand. So if two things work differently, even subtely, then in my view they shouldn't have the same name. I understand that this doensn't resolve the question at hand, but I'd have to read up a bit more on existing efforts before commenting on any of the proposed solutions. |
I think the best/easiest case is when there is nothing there at all, when there's nothing to explain. And removing u-naming is a way to simplify things such that there's no longer any difference to CPython and hence nothing to document/teach. Of course that's not the whole story because we still need a way to override built-in modules, so there will be something extra and something to document/teach. But for most use cases / most users / 80% of the time / to first order, things should just behave like CPython without having to worry about differences. At the moment it feels like it's the other way around, you need to first learn about u-naming before you can do anything. |
Sorry if I've missed this in the discussion, but did you consider the case where a frozen module adds bits/extends a built-in module ? For example if |
Our native apis are all documented as python stubs now. Folks have been adding type annotations now. See shared-bindings in the repo. (On my phone now)
…On Sat, Aug 6, 2022, at 4:47 PM, David Lechner wrote:
> Is there a particular ide / autocomplete package you're using currently?
>
We're using Pylance(based on Pyright) in VSCode and also Jedi.
> A proper micropython autocomplete setup is something I'm keen to get working.
>
Microbit forked Pyright to make it work in the browser and with their MicroPython API. You can see it in action at https://python.microbit.org/v/beta. Making this work in general for MicroPython with some sort of manifest system as you have describe would be ideal IMHO.
—
Reply to this email directly, view it on GitHub <#9018 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAM3KMSC3IYPZRBNENJPJTVX32QRANCNFSM55V4ZESA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
I maintain similar stubs for MicroPython. Most of them are generated automatically from the repo, and try to consider variations across, ports and boards. |
See #11456 which opens another option for extending built-ins from Python. |
#11456 brought me here: I had no idea what "u"modules were for and have been using them interchangeably and passing on that code smell to whoever uses/learns-from our examples. (Granted, my existence in mostly the make-everything-a-C-module space of MicroPython does not lend itself well to finding and understanding these quirks, which is why I'm lurking the GitHub and trying to expand my knowledge.) It's pretty easy to see the extent of this across our MicroPython examples: https://github.com/search?q=repo%3Apimoroni%2Fpimoroni-pico+%22import+u%22&type=code Granted not all of these are a problem. But it does make In our case, I guess I should raise an issue/PR to switch from |
I think Damien's comment #9018 (comment) is about exactly this point -- we should just make it simple and obvious.
There's an additional piece of historical context (and a third function of the u-prefix) there which was that it disambiguated the package name on PyPI. This is no longer an issue. There is an open issue to rename We should do the same thing for
Yep, see the official guidance here: Specifically: "Other than when you specifically want to force the use of the built-in module, we recommend always using |
See #9069 for an implementation of this. In the end I went with a fifth option which doesn't change any behavior but still allows removes the u-prefix from the module objects.
This means that there should be no user-visible change other the output of Later we can consider removing the handling for |
U-module renaming, deflate module, IDF 5, board variants and Pico-W BLE This release of MicroPython sees the renaming of built-in modules to remove the u-prefix, a new deflate module with optional compression support, the introduction of board variants, switching of the esp32 port to use IDF 5 together with improved heap management, support for BLE on RPi Pico W boards, and STM32H5xx support. The project is also now using codespell and ruff to improve code quality. New boards added in this release are: ARDUINO_NANO_ESP32 and UM_NANOS3 (esp32 port), ADAFRUIT_METRO_M7 (mimxrt port), ARDUINO_PORTENTA_C33 and VK_RA6M5 (renesas-ra port), ADAFRUIT_METRO_M4_EXPRESS (samd port), NUCLEO_L4A6ZG and STM32H573I_DK (stm32 port). The renaming of built-in modules to remove the u-prefix -- for example utime becomes time, uasyncio becomes asyncio -- is done to improve compatibility with CPython and eliminate confusion about whether to import the u-version or the non-u-version of the name. Now, one should just always import the non-u-version and no longer think about u-naming. The only remaining module with a u-prefix is uctypes because it is not compatible with the CPython ctypes modules. The following modules are still available via their u-names for backwards compatibility: array, asyncio, binascii, bluetooth, collections, cryptolib, errno, hashlib, heapq, io, json, machine, os, platform, random, re, select, socket, ssl, struct, time, websocket. These modules (except for asyncio) are also extensible and can be overridden by a file with the same name, eg time.py. To force the import of a built-in, one must first clear (and subsequently restore) sys.path; the ability to write to the sys.path attribute has also been implemented in this release. For further information see the discussion at micropython#9018. Furthermore, importing itself has been tweaked and optimised, and importing of built-ins no longer touch the filesystem, which makes a typical built-in import take ~0.15ms rather than 3-5ms. For modules that fail to import, they are now removed from sys.modules, allowing the import to be attempted again. This required adding "NLR jump callbacks" to efficiently run some code if an exception is raised. Additional improvements to the core runtime include support for conversion specifiers in f-strings (e.g. !r), speeding up of incremental GC cycles by tracking the last used block, addition of a new MICROPY_GC_SPLIT_HEAP_AUTO "auto grow heap" mode and support for pad bytes in struct format. The documentation, examples and test have also seen general improvements and additions. For the extended modules, the zlib C module has been removed and replaced with a new MicroPython-specific deflate module and DeflateIO class that is optimised to provide efficient streaming compression and decompression. The zlib (and gzip) modules are now implemented in pure Python on top of the deflate module. The timeq module has been removed, it existed only for a previous version of (u)asyncio and is no longer used. In the ssl module, SSLContext has been added to be more compatible with CPython. The select module now supports using system/POSIX poll when possible and the unix port now uses this implementation instead of its own one. That means the unix port can now select/poll on custom Python objects that implement the appropriate ioctl. The socket module now supports SO_BROADCAST, and sys.std*.buffer objects now exist on unix. There is also a new esp-hosted network driver for external ESP32-based network coprocessors. mpy-cross now allows reading source files from stdin and writing compiled code to stdout. The esp8266 and esp32 ports add support for the Espressif ESP-NOW protocol, and the ability to set/get the power saving mode of the WLAN hardware. The esp8266 port adds board-variant support, combining all boards into a single ESP8266_GENERIC with FLASH_512K, FLASH_1M and OTA variants. The esp32 port has now switched exclusively to ESP-IDF 5, and all existing components have been updated to work with this new IDF (except esp32.hall_sensor() which has been removed). The FSM ULP has been enabled for S2 and S3 chips, sockets now support SO_BINDTODEVICE, and board-named pins and the Pin.board dict have been implemented. The MicroPython heap has been reworked on this port to support the large variety of memory configurations: it now starts at 64kbytes and automatically grows as needed, with new segments allocated from the IDF heap. This means that boards with SPIRAM have much faster GC collection times if only a small amount of RAM is used, all available RAM can be used if needed, and the IDF has access to enough RAM for things like SSL sockets. For more information see micropython#12141. The esp32 port has also had its generic boards consolidated and renamed to ESP32_GENERIC, ESP32_GENERIC_C3, ESP32_GENERIC_S2 and ESP32_GENERIC_S3, and some now have variants such as SPIRAM. The mimxrt port has fixes and improvements to PWM and Pin.irq, as well as support for UART hardware flow control and UART.deinit. It has also seen integration of WiFi via the CYW43 driver, and Bluetooth via NimBLE. The mbedTLS bindings have enabled time validation of certificates. The renesas-ra port has changed board names to match the product name, updated to use FSP v4.4.0, added support for: RA6M5 MCUs, machine PWM, DAC, RNG and SDCard classes, TinyUSB, Bluetooth via NimBLE, networking via lwIP, and mbedTLS. Bluetooth support has also been added to the rp2 port for the Pico W board. And the RPi boards have been renamed from PICO to RPI_PICO, and PICO_W to RPI_PICO_W. Lightsleep has been fixed on this port so it works while WiFi is powered off, and time.time_ns() now has microsecond resolution. The samd port sees the addition of SPI and QSPI flash drivers to support filesystems on external flash. The stm32 port add support for STM32H5xx MCUs, basic support for the OCTOSPI peripheral, and USB support for STM32L1xx MCUs. New functions have been added to the stm module to support the sub-GHz radio on STM32WL55. In micropython-lib, an extensive LoRa module has been added along with drivers for SX126x and SX127x chipsets, and the STM32WL55. This module supports both synchronous and asynchronous (asyncio) mode. Also, as part of the u-module renaming, urequests has been renamed to requests (but for backwards compatibility "import urequests" still works for now). The change in code size since the previous release for various ports is (absolute and percentage change in the text section): bare-arm: +192 +0.340% minimal x86: +310 +0.169% unix x64: +4784 +0.610% stm32: -524 -0.134% cc3200: +280 +0.154% esp8266: +8016 +1.151% esp32: +112133 +7.293% mimxrt: +3624 +1.015% renesas-ra: -2184 -0.348% nrf: +616 +0.332% rp2: +1920 +0.595% samd: -7904 -2.953% The changes that dominate these numbers are: - bare-arm, minimal, cc3200, nrf: NLR jump callbacks and support for extensible modules - unix: update of mbedTLS to v2.28.1, support for polling Python objects in select module - stm32: removal of the timeq module - esp8266: addition of the espnow module - esp32: switching to ESP-IDF 5 - mimxrt: UART features, time validation of SSL certificates - renesas-ra: move to FSP v4.4.0, remove timeq module - rp2: machine.PWM enhancements - samd: drop support for VfsLfs1 Performance is effectively unchanged since the previous release. Note that this is the last release to use the current versioning scheme for nightly/unstable builds, whereby a build between releases is versioned as v1.20.0-<num>-g<hash> (following the release of v1.20.0). Moving forward, nightly builds will now be called preview builds and be versioned with the next release number. For example, if the last release was v1.21.0 then preview releases will be of the form v1.22.0-preview.<num>.g<hash>. For discussion see micropython#12127. Thanks to everyone who contributed to this release: Adam Green, Alexander Wilde, algonell, Andrew Leech, Andy Piper, Angus Gratton, Armin Brauns, brave ulysses, Brett Cannon, Brian 'redbeard' Harrington, Carlosgg, Chris Wilson, Christian Clauss, Damien George, Damien Tournoud, Daniël van de Giessen, David Grayson, David Lechner, David Yang, dotnfc, Duncan Lowther, Elecia White, elibdev, Elvis Pfutzenreuter, Felix Dörre, Francis Dela Cruz, Glenn Moloney, glenn20, iabdalkader, Ihor Nehrutsa, Jared Hancock, Jim Lipsey, Jim Mussared, Jon Nordby, Jonas Scharpf, Jos Verlinde, Kwabena W. Agyeman, Luca Burelli, marble, Mark Grosen, mbedNoobNinja, mcskatkat, Mingjie Shen, Mirko Vogt, Nicholas H. Tollervey, Oliver Joos, Ondrej Wisniewski, patrick, Peter Harper, Phil Howard, Philipp Ebensberger, Rene Straub, robert-hh, Sebastian Romero, Seon Rozenblum, stephanelsmith, stijn, Takeo Takahashi, Thomas, Tobias Thyrrestrup, UnexpectedMaker, Victor Rajewski, vsfos, Wang Xuancong, Wanlin Wang, Wilko Nienhaus, Wind-stormger, Yaroslav Halchenko, Yilin Sun, Yuuki NAGAO. The work done in this release was funded in part through GitHub Sponsors, and in part by George Robotics, Planet Innovation, Espressif, Arduino, LEGO Education and OpenMV. What follows is a detailed list of changes, generated from the git commit history, and organised into sections. Main components =============== all: - fix spelling mistakes based on codespell check - fix strings with backslash by using raw string literals - fix various Python coding inconsistencies found by ruff - fix cases of Python variable assigned but never used - rename MP_QSTR_umodule to MP_QSTR_module everywhere - rename mp_umodule*, mp_module_umodule* to remove the "u" prefix - rename mod_umodule*, ^umodule* to remove the "u" prefix - rename UMODULE to MODULE in preprocessor/Makefile vars - rename *umodule*.h to remove the "u" prefix - rename *umodule*.c to remove the "u" prefix - use MP_REGISTER_EXTENSIBLE_MODULE for overrideable built-ins - replace all uses of umodule in Python code - remove the zlib module - remove query-variants make target - add missing imports for micropython.const - add Black configuration section to pyproject.toml - add ruff to pre-commit - CODECONVENTIONS: require that commits be signed-off by the author py core: - ringbuf: implement put_bytes/get_bytes functions - parse: fix build when COMP_CONST_FOLDING=0 and COMP_MODULE_CONST=1 - compile: remove over-eager optimisation of tuples as if condition - stackctrl: add gcc pragmas to ignore dangling-pointer warning - gc: make improvements to MICROPY_GC_HOOK_LOOP - obj: remove mp_generic_unary_op() - objslice: ensure slice is not hashable - objdict: fix __hash__ for dict_view types - objarray: disallow memoryview addition - objstr: return unsupported binop instead of raising TypeError - runtime: if inplace binop fails then try corresponding normal binop - change MP_UNARY_OP_INT to MP_UNARY_OP_INT_MAYBE - obj: accept user types in mp_obj_get_int_maybe - objint: allow int() to parse anything with the buffer protocol - builtinimport: handle empty sys.path correctly - builtinimport: optimise sub-package loading - builtinimport: allow builtin modules to be packages - objmodule: don't use sys.modules to track a builtin __init__ - nlrsetjmp: use MP_NLR_JUMP_HEAD macro to simplify code - nlr: remove commented-out debugging code - nlr: implement jump callbacks - use nlr jump callbacks to optimise compile/execute functions - builtinimport: remove partially-loaded modules from sys.modules - builtinimport: remove weak links - makemoduledefs.py: add a way to register extensible built-in modules - objmodule: add a table of built-in modules with delegation - objmodule: workaround for MSVC with no module delegation - mpconfig: enable module delegation if sys needs it - modsys: allow sys.path to be assigned to - mkrules.mk: automatically configure frozen options when manifest set - parsenum: fix typo in #endif comment - nlraarch64: fix dangerous use of input register - makemoduledefs.py: fix declaring multiple module delegations - makemoduledefs.py: automatically declare delegation attr functions - lexer: allow conversion specifiers in f-strings (e.g. !r) - mkrules.mk: allow $(AFLAGS) to set flags to $(AS) - compile: fix async for's stack handling of iterator expression - builtinimport: fix built-in imports when external import is disabled - stream: add mp_stream___exit___obj that calls mp_stream_close - runtime: always initialise sched_state in mp_init - mpconfig: add MICROPY_PY_PLATFORM, enabled at extra features level - gc: speed up incremental GC cycles by tracking the last used block - gc: apply some code formatting cleanup - gc: add new MICROPY_GC_SPLIT_HEAP_AUTO "auto grow heap" mode - profile: remove the requirement to disable MICROPY_COMP_CONST - mpconfig: enable SSL finalizers if finalizers are enabled - objstr: fix `str % {}` edge case - modstruct: support pad bytes in struct format - dynruntime.h: implement MP_OBJ_NEW_QSTR - modthread: return thread id from start_new_thread() - malloc: fix DEBUG_print() args in m_realloc_maybe - runtime: add helpers to call a general function on nlr jump callback - parse: always free lexer even if an exception is raised - persistentcode: always close reader even if an exception is raised - gc: add "max new split" value in result of gc.mem_free() - nlrx64: mark nlr_push() as naked function when possible - mkrules.mk: don't strip binary if STRIP variable is unset - change ifdef DEBUG_PRINT to if DEBUG_PRINT - lexer: add missing initialisation for fstring_args_idx extmod: - utime_mphal: provide a general mktime function - modutime: provide a generic time module - machine_pwm: remove PWM_INIT and PWM_DUTY_U16_NS config options - network_cyw43: add power management constants - vfs_lfsx: fix offset used before range check - extmod.mk: suppress deprecated-non-prototype warning - moduos: move os.sync() into extmod/moduos.c - modtimeq: remove timeq module - btstack: add cmake support for BTstack - btstack: fix marking of static addresses in set_random_address - asyncio: rename uasyncio to asyncio - asyncio/uasyncio.py: add backwards-compatible uasyncio alias - modbinascii: fix buffer length error - update to support mbedtls 3.x - modplatform: set MICROPY_PLATFORM_ARCH on riscv platforms - modbtree: undefine queue macros before including berkeley-db - modssl: add SSLContext class - moddeflate: add deflate module providing the DeflateIO class - modssl_mbedtls: reference SSLContext from SSLSocket - vfs_posix_file: add poll support for missing ERR,HUP,NVAL values - modselect: abstract out a poll_set_t struct and functions - modselect: factor low-level polling code into common function - modselect: add optimisation to use system poll when possible - modselect: remove undocumented support for flags arg to poll - modssl_mbedtls: reject ioctls that are not supported - modssl_mbedtls: fix ioctl of a socket in closed/error state - modselect: properly track number of poll objects that are fd's - modssl_mbedtls: clear sock member if error creating SSLSocket - moddeflate: change default window size - vfs_posix_file: fix flush handling on macOS - vfs_posix_file: implement sys.std*.buffer objects - modlwip: add support for SO_BROADCAST socket option - modsocket: add support for SO_BROADCAST socket option - modssl_mbedtls: call func psa_crypto_init if PSA is used - modssl_mbedtls: ignore err ERR_SSL_RECEIVED_NEW_SESSION_TICKET - modlwip: fix setting of IP option SOF_BROADCAST - network_esp_hosted: add ESP-Hosted networking interface - modssl_axtls: only close underlying socket once if it was used - asyncio/event.py: fix ThreadSafeFlag.ioctl return - btstack/btstack_hci_uart: trigger a poll after UART data is sent - asyncio/stream.py: fix cancellation handling of start_server - modnetwork: increase max hostname length to 32 - modnetwork: forward if.config(hostname) to network.hostname - vfs_posix_file: fix flush handling in msvc builds shared: - upytesthelper: fix spelling of "default" - libc/printf: fix stdout destination for putchar and puts - tinyusb: avoid symbol clash on targets with external TinyUSB - tinyusb: support HS endpoint sizes - netutils/dhcpserver: reply on correct netif drivers: - cyw43: make the CYW43 Bluetooth HCI driver more portable - esp-hosted: add host driver for ESP-Hosted firmware - ninaw10/nina_bt_hci: make some minor fixes to HCI driver - esp-hosted: fix pin IRQ - esp-hosted: fix MTU size - esp-hosted: add support for WiFI LED activity indicator mpy-cross: - allow specifying source files starting with - - allow reading from stdin and writing to stdout - when reading from stdin, write output to stdout - allow specifying stdin as input without -- - fix source file name in file-not-found error lib: - mbedtls_errors: update error list for current version of mbedtls - mbedtls: update to mbedtls v2.28.1 - mbedtls_errors: add esp32-specific mbedtls error file - mbedtls_errors: update patch and error list for new mbedtls - mbedtls: update to mbedtls v2.28.3 - fsp: update FSP for renesas-ra to the latest version v4.4.0 - cyw43-driver: update driver to latest version v1.0.1 - btstack: update to v1.5.6.2 - pico-sdk: update to version 1.5.1 - stm32lib: update library for H5 v1.0.0 - oofatfs: fix speculative read in create_name - uzlib: add memory-efficient, streaming LZ77 compression support - uzlib/lz77: always use separate history buffer - uzlib/defl_static: implement some code size improvements - uzlib: clean up tinf -> uzlib rename - uzlib: combine zlib/gzip header parsing to allow auto-detect - uzlib/tinflate: implement more compact lookup tables - uzlib/defl_static: optimize zlib_start/finish_block - uzlib: add a source_read_data var to pass to source_read_cb - tinyusb: update to the most recent master - protobuf-c: add protobuf-c library - cyw43-driver: update driver to latest version v1.0.2 - micropython-lib: update submodule to latest Support components ================== docs: - reference: remove double 'are' in glossary - update the PWM examples based on recent API improvements - samd: make use of pin names more consistent in examples - reference/mpyfiles: add release info on v6.1 - library/espnow: update espnow docs for WLAN.config(pm=x) options - develop/porting: add missing code to example main.c and Makefile - reference/speed_python: remove 4-arg limit for viper - mimxrt: add the pin-out for the Adafruit Metro M7 board - samd: add the pin-out for the Adafruit Metro M4 Airlift board - library/index: update built-in extension docs - reference/packages: add GitHub repo to package example dependency - reference/mpremote.rst: extend the mpremote guide - library/index: update docs after umodule rename - rename uasyncio to asyncio - esp32: update esp32 docs based on IDF v5 changes - library/ssl: add documentation for SSLContext - esp32/quickref: add LAN example for WT32-ETH01 version 1.4 - library/deflate: add docs for deflate.DeflateIO - develop/gettingstarted: clarify submodule initialization - develop/gettingstarted: update ARM package list - library/neopixel: change link to a micropython-lib reference - library/platform: add docs for the platform library - library/network: clarify network.hostname() behaviour - esp32/tutorial: add example for pin access via registers - library/esp32: update ESP32 idf_heap_info docs to match behaviour - library/gc: clarify mem_alloc and mem_free only for Python heap - conf.py: add sphinxcontrib.jquery to extensions - add requirements.txt file with dependencies for Sphinx - change remaining "urequests" references to "requests" examples: - usercmodule: add a sub-package example - natmod: rename umodule to module - hwapi: rename uasyncio to asyncio - natmod/deflate: add deflate as a dynamic native module - mark asm, pio, etc. as noqa: F821 (undefined-name) - hwapi: add missing import for 96Boards Carbon example - bluetooth: raise ValueError when advertising data is too large - bluetooth: link to aioble in BLE examples - natmod: add features4 as a class definition example - unix/machine_bios.py: fix typo tests: - run-tests.py: ensure correct cwd for mpy tests - basics: add more tests for hashing of various types - basics: remove __index__ and __inv__ from special methods tests - import/builtin_ext.py: add test for built-in module override - import/import_pkg9.py: add test for subpackage attribute - replace umodule with module everywhere - run-multitests.py: don't allow imports from the cwd - run-perfbench.py: don't allow imports from the cwd - run-natmodtests.py: don't allow imports from the cwd - float: test domain errors for more combos of args to math funcs - rename uasyncio to asyncio - extmod/uctypes_array_assign_le: fix buffer - extmod/framebuf: fix buffer size issues - extmod: add tests for ssl.SSLContext - extmod: add test for passing cadata into ssl.wrap_socket() - extmod: add deflate.DeflateIO tests - extmod: add coverage tests for select module - extmod: skip select/socket tests if they can't create UDP socket - extmod/select_poll_eintr.py: improve robustness of test - misc/sys_settrace_features.py: fix to run on newer CPython - unix/mod_os: make os.system() test work on windows - run-tests.py: capture output of stderr when running on CPython - multi_net: increase asyncio tests timeouts - stress/bytecode_limit.py: reverse order of cases - float/float_format_ints.py: put power-of-10 test in separate file - extmod/deflate_decompress.py: skip test when not enough memory - extmod/ssl_cadata.py: skip test on axtls - float/math_domain.py: tweak test to also pass with obj-repr-C - extmod/vfs_fat_finaliser.py: tweak test so files are collected - README: document ./run-internalbench.py - run-internalbench.py: remove old CPython reference - multi_net/ssl_cert_rsa.py: update test certificate - extmod/asyncio_threadsafeflag.py: update for unix select tools: - pyboard.py: rename ProcessPtyToTerminal member "ser" to "serial" - mpremote: remove unused import of serial - pyboard.py: import serial.tools.list_ports - pyboard.py: import errno to fix undefined name in PyboardError - manifestfile.py: fix license capturing - mpremote: add repl option to escape non-printable characters - pydfu.py: use getattr to retrieve getargspec function - mpremote: add `sleep` command - mpremote: allow terminator for shortcut commands - mpremote: add `rtc` commands to get and set the RTC - mpremote: handle `cp` without destination - mpremote: detach mpremote from pyboard.py - mpremote: fix use of stdout_write_bytes function - mpremote: fix exec_ -> exec in commands.py - autobuild: update auto-build code to build esp32 port with IDF v5 - autobuild: add support for application .bin files for esp32 - mpy-tool.py: use isinstance() for type checking - codeformat.py: skip formatting ESP-IDF managed components - codeformat.py: use pyproject.toml for black config - mpremote: make soft-reset count as an action - autobuild: automatically build all variants for each board - mpy_ld.py: pre-declare some local variables to appease linter - mpy-tool.py: ignore linter failure in Python 2 compatibility code - mpy_ld.py: support more complex rodata sections - metrics.py: fix nrf and rp2 board names after renaming - autobuild: include .bin firmware in renesas-ra build output - autobuild/build-downloads.py: verify standard features - mpremote: add support for rfc2217, serial over TCP - metrics.py: fix esp32 and esp8266 board names after renaming - change remaining "urequests" references to "requests" CI: - ci.sh: build both SAMD21 and SAMD51 boards as part of samd CI - ci.sh: add functions to check code spelling using codespell - ci.sh: add mimxrt and samd ports to code size build - ci.sh: build PICO_W board as part of rp2 CI - ci.sh: add a H5 board to stm32 CI build - ci.sh: add ARDUINO_PORTENTA_C33 to RA CI build - workflows: fetch full history for mpremote workflow - workflows: add spell check to code formatting workflow - workflows: add GitHub Action to lint Python code with ruff - workflows: force use of Ubuntu-20.04 for unix 32-bit builds - workflows: update esp32 CI to use IDF v5.0 - workflows: bump actions/checkout from 3 to 4 The ports ========= all ports: - use extmod version of mktime instead of port-specific one - remove os.sync() implementation from stm32 and renesas-ra - enable os.sync() for esp32, esp8266, rp2, mimxrt, samd51 - standardise docs link in help text - in machine_i2s.c, rename uasyncio to asyncio - simplify board feature tags in board.json - */boards/*/board.json: remove "id" field - restrict board.json to standard features - rename Arduino board LED pins to be consistent bare-arm port: no changes specific to this component/port cc3200 port: - mods/modutime: use extmod version of time module - tools: fix exception raised on process failure - Makefile: build firmware.zip embed port: no changes specific to this component/port esp8266 port: - modutime: use extmod version of time module - add support for the Espressif ESP-NOW protocol - machine_pwm: implement duty_u16() and duty_ns() for consistency - add support to set/get power saving mode of WLAN - change network.WLAN from a function to a type - allow Ctrl-C to interrupt the corrupt-fs while loop - machine_pin: accept an integer argument to mp_obj_get_pin_obj - add board variant support - boards/ESP8266_GENERIC: add image filename - boards: make sure modespnow.o is placed in irom0 - boards/ESP8266_GENERIC: remove urllib from the 2MiB manifest esp32 port: - modutime: use extmod version of time module - add support for the Espressif ESP-NOW protocol - add support to set/get power saving mode of WLAN - change network.WLAN from a function to a type - boards: add some missing board configs for two UM boards - esp32_ulp: enable FSM ULP for S2 and S3 chips - uart: use xtal as UART clock source on S3 and C3 - modespnow: change name of buffer size config option to "rxbuf" - CMake: change PROJECT_DIR to CMAKE_CURRENT_LIST_DIR - esp32_ulp: fix ULP (FSM) support for S2 and S3 - allow Ctrl-C to interrupt the corrupt-fs while loop - switch from UART driver to UART HAL - ppp_set_auth: add pppapi_set_auth from ESP-IDF - modesp32: remove esp32.hall_sensor function - update port to support IDF v5.0.2 - in recv_cb, get espnow rssi from recv_info->rx_ctrl - network_wlan: wait for WIFI_EVENT_STA_START after activating - Makefile: provide more IDF shortcuts - boards: change SDK config parameters from deprecated to new ones - modules/inisetup.py: format partition as FAT if its label is ffat - machine_uart: always select a source_clk value in UART config - re-enable mDNS after move to IDF v5.0.2 - boards/GENERIC_OTA: enable silent checks to reduce firmware size - network_wlan: wait for STA/AP START/STOP event in wlan.active - machine_timer: switch from legacy driver to timer HAL - machine_pin: add a pin-find func and use it in machine_pin_get_id - use always machine_pin_get_id for getting a Pin id - add support for board-named pins and the Pin.board dict - collect properties from IDF-managed components as well - modmachine: add generic machine.bootloader() - usb: add custom TinyUSB callback support - boards/ARDUINO_NANO_ESP32: add support for Arduino Nano ESP32 - CMakeLists: enable multiple extra component directories in build - boards/ARDUINO_NANO_ESP32: fix deploy instructions - main: remove unused mbedtls debug function - machine_wdt: allow feeding WDT from threads - machine_hw_spi: fix access of SPI(2) - machine_hw_spi: remove unnecessary duplicate SPI pin defaults - machine_hw_spi: remove SPI host renaming for C3 and S3 variants - machine_hw_spi: check for valid SPI id in constructor, not init - boards: remove references to the IDF version in board.md files - README: specify that only IDF v5.0.2 is supported - allow malloc() to allocate from SPIRAM - enable automatic Python heap growth - gccollect: make level arg volatile to force recursive function - Makefile: implement `make submodules` to match other ports - boards/GENERIC: merge with GENERIC_{SPIRAM,OTA,D2WD,UNICORE} - boards/GENERIC_C3: merge with GENERIC_C3_USB - boards/GENERIC_S2: merge with ESP32_S2_WROVER - boards/GENERIC_S3: merge with GENERIC_S3_{SPIRAM,SPIRAM_OCT} - partitions.csv: rename to partitions-4MiB.csv - use uppercase variant names - Makefile: append board variant to BUILD - rename GENERIC* boards to ESP32_GENERIC* - boards/ESP32_GENERIC_C3: enable UART REPL - modsocket: add support for SO_BROADCAST socket option - modnetwork: add support for SO_BINDTODEVICE socket option - support JTAG console, free up UART - machine_uart: release GIL for blocking reads - boards: add pins.csv to UM boards and other minor changes - boards/UM_NANOS3: add new UM NanoS3 board - network_ppp: block after deleting task - boards/ARDUINO_NANO_ESP32: clarify recovery instructions - boards/UM_FEATHERS3: fix I2C pins in pins.csv - skip validation of image on boot from deepsleep - machine_pin: fix null pointer access in machine_pin_find - mphalport: add function/line/file info to check_esp_err exception - fix Partition.writeblocks() partial write corruption - boards: fix VBAT voltage calculation for UM S3 boards - boards: add bootloader rollback support for all builds - main: allow a board to override the MicroPython task stack size - boards/ARDUINO_NANO_ESP32: use Arduino USB IDs - boards/manifest.py: freeze aioespnow into firmware by default mimxrt port: - modutime: use extmod version of time module - machine_pwm: start PWM only if freq and duty are set - flash: separate low level driver code from flash object - mpconfigport: add back lost uos.urandom() - add missing UART defintion and remove obsolete config - machine_spi: ignore transfers with len=0 - machine_pin: perform full configuration in machine_pin_set_mode - sdcard: fix GCC 13 build error with sdcard_cmd_set_bus_width - led: add support for up to four LEDs - boards/ADAFRUIT_METRO_M7: add Adafruit Metro M7 board definition - machine_pwm: fix freq change, PWM print, and error checks - Makefile: use a specific fsl_flexspi_nor_boot.c for mimxrt1062 - machine_pin: fix bug when Pin.irq is called without a handler - hal/pwm_backport: fix 0 and 65536 edge cases of PWM's duty_u16 - machine_uart: add uart.deinit method and machine_uart_deinit_all - machine_uart: add support for UART hardware flow control - boards: add support for GPIO control of SNVS pins - hal: make flash clock frequency configurable - fix UART RTS/CTS assignments for the OLIMEX and Adafruit boards - machine_pin: extend pin configuration functions - sdio: add SDIO driver - integrate support for WiFi via the CYW43 driver - integrate Bluetooth support with NimBLE bindings - irq: move all IRQ related definitions to dedicated header - machine_uart: fix and complete UART.deinit and uart_deinit_all - boards: fix use of MICROPY_HW_SDRAM_AVAIL in MIMXRT1176.ld - machine_uart: support slow baud rates for UART - machine_uart: add a helper function to change the baudrate - sdio: move config guard so headers are only included if used - Makefile: update to work with latest TinyUSB - mpconfigport: don't override parse chunk alloc - sdio: add support for the 117x series - mimxrt_sdram: allow boards to override the default SDRAM config - Makefile: enable the FSL USDHC for supported MCU series - remove SDCARD Makefile config option - mpbthciport: enable flow control for BT HCI UART - mbedtls: enable certificate validity time validation - machine_uart: set the UART clock to a fixed 40MHz value - boards/MIMXRT1176_clock_config: fix comments about UART clocks - boards: fix naming of SD-card config option - mpbthciport: allow disabling UART flow control for BLE - machine_rtc: improve the RTC init at boot minimal port: no changes specific to this component/port nrf port: - modules/utime: use extmod version of time module - boards: rename all nRF boards to use uppercase pic16bit port: no changes specific to this component/port powerpc port: - mpconfigport: don't override parse chunk alloc qemu-arm port: no changes specific to this component/port renesas-ra port: - change MICROPY_HW_BOARD_NAME definition to product name - modutime: use extmod version of time module - update boards and ra directory files to support FSP v4.4.0 - add a macro definition to avoid compile error of FSP v4.4.0 - irq: fix typo in comment about IRQ priorities - consolidate hal_entry.c code and remove hal_entry() func - boards/make-pins.py: fix PA/PB pins support - consolidate all fsp_cfg header files to one location - support changing baudrate for UART - add support for RA6M5, and add machine PWM, DAC, SDCard - boards/VK_RA6M5: add new board definition - remove duplicate machine module from constants list - machine_spi: consistently use machine_pin_find to get pin - boards: remove unreachable code in make-pins.py - Makefile: generate binary firmware output - add TinyUSB support - add Bluetooth support using NimBLE - add RNG driver - add networking support using lwIP - add mbedTLS support - fsp_cfg: add common FSP config files - boards/ARDUINO_PORTENTA_C33: add support for Portenta C33 - boards/ARDUINO_PORTENTA_C33: update WiFi config - tune lwip buffers and timing to improve network performance rp2 port: - modutime: use extmod version of time module - machine_pwm: enable keyword args in constructor and add init method - machine_pwm: add support for inverting a PWM channel output - machine_pwm: add duty_x() checks and return 0 if PWM is not started - make rp2_state_machine_exec accept integers - CMakeLists: allow relative MICROPY_BOARD_DIR when invoking cmake - mphalport: only use CYW43 MAC for WLAN0 interface - CMake: normalize MICROPY_PORT_DIR - add Bluetooth support via cyw43 - mpbthciport: cancel existing alarms - boards/PICO_W: enable Bluetooth Low Energy support - machine_pin: factor out pin-find code from machine_pin_make_new - use uppercase variant names - Makefile: append board variant to BUILD - rename PICO, PICO_W to RPI_PICO, RPI_PICO_W - machine_timer: fix printing of timer period - mpbthciport: switch to static scheduler nodes - mpbthciport: fix HCI UART config - mpconfigport: disable BLE locking when MICROPY_PY_BLUETOOTH enabled - boards/ARDUINO_NANO_RP2040_CONNECT: use standard HCI UART baudrate - modmachine: fix lightsleep while wifi is powered off - msc_disk: allow configuring the USB MSC inquiry response - README: fix name of RPI_PICO_W board - CMakeLists: enable debug symbols in all builds - implement time.time_ns with time_us_64 so it has us resolution samd port: - modutime: use extmod version of time module - machine_pwm: add init() method to PWM and simplify the PWM code - mpconfigport: drop support for SoftSPI max speed - boards/MINISAM_M4: update pins.csv for the Mini SAM M4 board - rearrange the MCU-specific loader files - mcu/samd51: enable MICROPY_ENABLE_EMERGENCY_EXCEPTION_BUF - boards: add default deploy instructions - main: fix sercom deinit ordering in soft-reset - modmachine: add machine.deepsleep as alias of machine.lightsleep - modmachine: make some machine classes configurable by #defines - boards/SEEED_WIO_TERMINAL: rename two pins starting with a digit - mcu: reduce the startup time after hard reset - machine_uart: add support for UART hardware flow control - boards/ADAFRUIT_METRO_M4_EXPRESS: add Metro M4 Express Airlift - boards: rename flash pins consistently for QSPI and SPI - samd_spiflash: add SPI flash driver and configure it accordingly - samd_qspiflash: add QSPI flash driver and configure it accordingly - adapt existing samd.Flash and integrate with (Q)SPI flash in boot - boards: extend the code size limit for boards with external flash - set the LFS type in mpconfigmcu.mk instead of mpconfigboard.mk - Makefile: move the math lib files from mpconfigmcu.mk to Makefile - Makefile: print memory region usage instead of totals - mpconfigport: enable DHT and onewire drivers on all MCUs/boards - mpconfigport: provide the platform module - modules/_boot.py: add /lib to sys.path - mpconfigport: don't override parse chunk alloc stm32 port: - modutime: use extmod version of time module - boards/ARDUINO_PORTENTA_H7: enable FDCAN - boards/stm32h723_af.csv: fix ADC AF definitions - mboot: fix alignment of packed final buffer - irq: fix typo in comment about priorities - usb: fix USB support on STM32G4 - boards/NUCLEO_G474RE: make it easier to enable USB - adc: add support for STM32L4A6 MCUs - boards/NUCLEO_L4A6ZG: add new board definition - help: don't include unavailable features in help() - help: exclude legacy entries from help, and adjust constant names - mpconfigboard_common: provide default spidev config - boards: use default spibdev config where appropriate - Makefile: pass relevant CPU flags to assembler - make-stmconst.py: support structs with names ending in _t - main: start UART REPL as early as possible - usbd_conf: treat G0 USB periph as MICROPY_HW_USB_IS_MULTI_OTG=0 - machine_adc: handle ADC resolution less than 8 bits on all MCUs - add initial support for H5 MCUs - boards: add ld, af.csv and hal_conf_base.h files for H5 MCUs - octospi: add preliminary support for OCTOSPI peripheral - boards/STM32H573I_DK: add H5 board definition files - add USB support for STM32L1 MCUs - modify RCC->APB2ENR directly instead of HAL API - modmachine: remove duplicate machine_timer_type declaration - modmachine: make machine_reset_cause_obj public - dac: fix dac.write_timed on G4 MCUs to use 32-bit DMA access - adc: fix ADC clock prescaler for G4 MCUs - adc: fix pyb.ADCAll.read_core_temp for G4 MCUs - adc: fix reading internal ADC channels on G4 MCUs - machine_adc: fix machine.ADC to work on G4 MCUs - adc: add workaround for ADC errata with G4 MCUs - adc: fix pyb.ADCAll.read_core_bat on G4 and L4 MCUs - qspi: allow qspi_write_cmd_data to write cmd with 1 data byte - mpconfigport: always define MICROPY_SOFT_TIMER_TICKS_MS - remove duplicate machine module from constants list - boards/B_L072Z_LRWAN1: add pin definitions for internal SX1262 - boards/B_L072Z_LRWAN1: lower default ROM level to "Core" - spi: add STM32WL5 SUBGHZ SPI peripheral - powerctrlboot: support STM32WL system clock from HSE+PLL - dma: fix DMA completion on WL55 boards - subghz: add STM32WL55 subghz radio interface to stm module - modstm: add MICROPY_PY_STM_CONST flag, clear it for STM32WL5 - use uppercase variant names - Makefile: append board variant to BUILD - mboot: fix fwupdate by replacing zlib with new deflate module - uart: fix UART timeout issue with low baudrate on G4 MCUs - timer: fix deadtime config on Advanced Timer peripherals - dma: add support for SPI1 DMA on H5 MCU's - octospi: add support for dual-line SPI interface - powerctrlboot: allow using PLL3 for USB clock on H5 MCU's - timer: fix use of TIM8 on H5 MCU's - uart: add support for UART10 when it's a USART - powerctrlboot: allow PLL1 Q and R outputs to be enabled on H5 - adc: fix STM32H5 support - adc: add support for STM32H5 ADC2 inputs - adc: optimize sampling time for G4, H5, L4 and WB MCUs - machine_adc: fix and improve STM32H5 support - dac: add STM32H5 DAC support, with dma_nohal implementation - boards: move includes to after defines in all hal_conf.h files - uart: generalise UART source clock calculation for H5 and H7 MCUs - dma: remove unbalanced ) - usbd_msc_interface: allow configuring the MSC inquiry response - boards/ARDUINO_GIGA: update board config - i2c: add support for I2C4 on H7 MCUs teensy port: no changes specific to this component/port unix port: - modutime: use extmod version of time module - Makefile: allow variants to add QSTR_DEFS - main: add NLR scope for checking module __path__ - README: fix Markdown link markup - modsocket: add poll support for missing ERR,HUP,NVAL poll values - modselect: remove unix-specific implementation of select module - modjni: fix build errors with type definitions and error strings - main: fix memory leakage if MICROPY_USE_READLINE is disabled webassembly port: - modutime: use extmod version of time module - make mp_js_do_str asynchronous - make mp_js_process_char asynchronous - replace typeof window check with ENVIRONMENT_IS_NODE flag windows port: - Makefile: allow variants to add QSTR_DEFS zephyr port: - modutime: use extmod version of time module
Add Adafruit Feather ESP32-C6 4MB Flash No PSRAM
We use the "u" prefix for built-in modules for several reasons (see #7499 (comment) for the full story). The primary reason though is to allow
foo.py
to exist on the filesystem and extend the built-inufoo
. This feature is called "weak links" because the namefoo
used to be a "weak link" toufoo
(i.e. the link is "broken" by having the filefoo.py
on the filesystem). Today the feature, when enabled, is automatic for any built-in namedufoo
.There's a few drawbacks:
import foo
is slower than necessary because it must search the filesystem to (usually) not findfoo.py
insys.path
only to eventually findufoo
in the builtins table.builtin.__import__
)import foo; print(foo)
you getufoo
. Alsohelp('modules')
lists everything asufoo
.import asyncio
, and also it should be possible to extend asyncio with optional features.Our goal here is usually just to provide "optional" implementations of functionality that aren't general-purpose enough to include in standard firmware, or possibly are just better suited to implementation in Python. (i.e. they could still be good candidates for being frozen?). But in general we're "filling in" missing functionality from the CPython equivalent module, and therefore there's no precedent for this in CPython. Note that CircuitPython does not currently enable the "weak link" feature and doesn't provide a way to extend built-in modules from Python.
I would argue that based on the reasons above, the "ufoo" mechanism doesn't support this use case particularly well, and the other historical reasons for the "u" prefix aren't compelling either. So let's say that in a future release (2.0?) that we remove all traces of the "u" prefix for built-in modules (i.e. rename all built-ins, remove the weak links feature, remove the last remaining traces from the documentation). This is obviously a hugely breaking change, would need to be done gradually.
So that means we need to solve what to do about extending built-in modules. There's four options that I can think of:
Don't allow this. It's worked so far for CircuitPython, and it's entirely possible that many MicroPython users have never needed/wanted it either (I know this isn't quite true).
Allow a way to "runtime patch" built-in modules. So instead of replacing "foo", you could import a module (e.g.
import foo_ext
) that would install/replace its own additional methods/classes into some built-in modulefoo
. This is currently impossible because the modules' globals tables are in ROM. We have a partial mechanism for this that is selectively enabled for thesys
module (to allow e.g.sys.ps1
), and also thebuiltins
module is special cased to allow this for all elements.It seems feasible to implement this for all built-in modules without too much RAM or runtime performance cost. It composes well -- for example we can provide a bunch of different
os_ext_foo
,os_ext_bar
,os_ext_baz
to provide different extensions to theos
module, and the user can choose which ones to install, and then "activate" them by importing them once at the top of main.py (or even boot.py). It's perhaps a bit awkward that you need this activation step though, but from that point onimport os
will have all the extensions ready.It also addresses the asyncio case. We can rename the frozen uasyncio, and then provide optional extensions which install themselves at runtime (except for asyncio it's simple because it's frozen not built-in so the modules are already mutable).
In more detail, the idea would be to have lazy-constructed
MP_STATE_VM(module_overrides)
, (either a single layer ofmodule+qstr -> obj
or two-layermodule->(qstr->obj)
) that is used preferentially for the module's globals dict. As a performance optimisation (especially for tab completion which does a lot of module lookups) we need a single bit somewhere to indicate that a module has had at least one override.from ufoo import *
approach. One simple example would befrom builtins.foo import *
orfrom micropython.foo import *
(this is a fairly straightforward mod toobjmodule.c
andbuiltinimport.c
to detect the "builtins." prefix and resolve the module viamp_module_get_builtin
). Another alternative is to make builtins available inmicropython.builtins
or something (this would also provide a mechanism to programatically query built-ins which is occasionally requested). Another way could be to empty sys.path before importing (although this doesn't currently work, an empty sys.path is equivalent to['']
).This doesn't solve the composability, or the fact that we still need to search the filesystem, or how to extend asyncio.
It also has a RAM cost for extending a builtin-module because the "from builtin.foo import *" needs to copy the entire globals dict of the built-in into the Python module that's replacing it.
builtin.__import__
. As mentioned above, it is actually technically possible in CPython to hook import, even for builtins, and then return an extended module. For example, in "os_ext_foo.py".then
This composes well, avoids the filesystem search for builtin import, works for asyncio,
but has a fairly high RAM cost for duplicating the globals table (potentially once for each extension)(edit: solution below), as well as the cost for the hook code.CC a few people who have been involved in this topic in the past: @mattytrentini @andrewleech @tannewt @jepler @dhalbert @dlech @stinos
The text was updated successfully, but these errors were encountered: