Thanks to visit codestin.com
Credit goes to github.com

Skip to content

RFC: Built-in module extending and removing weak links / umodules #9018

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jimmo opened this issue Aug 5, 2022 · 28 comments
Closed

RFC: Built-in module extending and removing weak links / umodules #9018

jimmo opened this issue Aug 5, 2022 · 28 comments
Labels
enhancement Feature requests, new feature implementations

Comments

@jimmo
Copy link
Member

jimmo commented Aug 5, 2022

We use the "u" prefix for built-in modules for several reasons (see #7499 (comment) for the full story). The primary reason though is to allow foo.py to exist on the filesystem and extend the built-in ufoo. This feature is called "weak links" because the name foo used to be a "weak link" to ufoo (i.e. the link is "broken" by having the file foo.py on the filesystem). Today the feature, when enabled, is automatic for any built-in named ufoo.

There's a few drawbacks:

  • import foo is slower than necessary because it must search the filesystem to (usually) not find foo.py in sys.path only to eventually find ufoo in the builtins table.
  • This is a CPython incompatibility (in CPython you cannot replace a built-in from Python via the filesystem... although you can hook builtin.__import__)
  • It doesn't work particularly well if you want to apply multiple extensions from different sources (i.e. you can't really compose this approach)
  • It's weird that when you write import foo; print(foo) you get ufoo. Also help('modules') lists everything as ufoo.
  • It's just generally confusing and difficult to document and explain. We are slowly improving this, but there's still a lot of code out there writing "import ufoo".
  • Weak links don't apply to frozen modules, so it doesn't align well with non-built-in-but-frozen-and-kind-of-like-builtin modules like "uasyncio". You should be able to write import asyncio, and also it should be possible to extend asyncio with optional features.

Our goal here is usually just to provide "optional" implementations of functionality that aren't general-purpose enough to include in standard firmware, or possibly are just better suited to implementation in Python. (i.e. they could still be good candidates for being frozen?). But in general we're "filling in" missing functionality from the CPython equivalent module, and therefore there's no precedent for this in CPython. Note that CircuitPython does not currently enable the "weak link" feature and doesn't provide a way to extend built-in modules from Python.

I would argue that based on the reasons above, the "ufoo" mechanism doesn't support this use case particularly well, and the other historical reasons for the "u" prefix aren't compelling either. So let's say that in a future release (2.0?) that we remove all traces of the "u" prefix for built-in modules (i.e. rename all built-ins, remove the weak links feature, remove the last remaining traces from the documentation). This is obviously a hugely breaking change, would need to be done gradually.

So that means we need to solve what to do about extending built-in modules. There's four options that I can think of:

  1. Don't allow this. It's worked so far for CircuitPython, and it's entirely possible that many MicroPython users have never needed/wanted it either (I know this isn't quite true).

  2. Allow a way to "runtime patch" built-in modules. So instead of replacing "foo", you could import a module (e.g. import foo_ext) that would install/replace its own additional methods/classes into some built-in module foo. This is currently impossible because the modules' globals tables are in ROM. We have a partial mechanism for this that is selectively enabled for the sys module (to allow e.g. sys.ps1), and also the builtins module is special cased to allow this for all elements.

It seems feasible to implement this for all built-in modules without too much RAM or runtime performance cost. It composes well -- for example we can provide a bunch of different os_ext_foo, os_ext_bar, os_ext_baz to provide different extensions to the os module, and the user can choose which ones to install, and then "activate" them by importing them once at the top of main.py (or even boot.py). It's perhaps a bit awkward that you need this activation step though, but from that point on import os will have all the extensions ready.

It also addresses the asyncio case. We can rename the frozen uasyncio, and then provide optional extensions which install themselves at runtime (except for asyncio it's simple because it's frozen not built-in so the modules are already mutable).

In more detail, the idea would be to have lazy-constructed MP_STATE_VM(module_overrides), (either a single layer of module+qstr -> obj or two-layer module->(qstr->obj)) that is used preferentially for the module's globals dict. As a performance optimisation (especially for tab completion which does a lot of module lookups) we need a single bit somewhere to indicate that a module has had at least one override.

  1. Provide a better way to force importing of a built-in to replace the existing from ufoo import * approach. One simple example would be from builtins.foo import * or from micropython.foo import * (this is a fairly straightforward mod to objmodule.c and builtinimport.c to detect the "builtins." prefix and resolve the module via mp_module_get_builtin). Another alternative is to make builtins available in micropython.builtins or something (this would also provide a mechanism to programatically query built-ins which is occasionally requested). Another way could be to empty sys.path before importing (although this doesn't currently work, an empty sys.path is equivalent to ['']).

This doesn't solve the composability, or the fact that we still need to search the filesystem, or how to extend asyncio.

It also has a RAM cost for extending a builtin-module because the "from builtin.foo import *" needs to copy the entire globals dict of the built-in into the Python module that's replacing it.

  1. Do this via builtin.__import__. As mentioned above, it is actually technically possible in CPython to hook import, even for builtins, and then return an extended module. For example, in "os_ext_foo.py".
import builtins, sys

from os import *

# Add os.foo
def foo():
    print('hello foo')

x = builtins.__import__
def _hook(name, globals, locals, fromlist, level):
    if name == "os":
        return sys.modules[__name__]
    return x(name, globals, locals, fromlist, level)

builtins.__import__ = _hook

then

import os_ext_foo

import os
os.foo()

This composes well, avoids the filesystem search for builtin import, works for asyncio, but has a fairly high RAM cost for duplicating the globals table (potentially once for each extension) (edit: solution below), as well as the cost for the hook code.

CC a few people who have been involved in this topic in the past: @mattytrentini @andrewleech @tannewt @jepler @dhalbert @dlech @stinos

@jimmo jimmo added the enhancement Feature requests, new feature implementations label Aug 5, 2022
@jimmo
Copy link
Member Author

jimmo commented Aug 5, 2022

Some other misc thoughts that don't quite fit into the above:

  • It would be good to move towards only providing a subset of CPython functionality in our built-in modules, and to that end we should clarify exactly what the rules are. Entirely new methods/classes -- not ok. Different functionality -- not ok. Adding extra kwargs to existing methods -- maybe ok?
  • Should we find new homes for existing non-CPython extensions (e.g. time.ticks* --> ticks.ticks*?, os.mount --> storage.mount?, asyncio.sleep_ms --> uhh... uasyncio.sleep_ms?) and how long should we keep aliases to the existing ones.
  • Is it important to be able to replace as well as extend functions in existing modules. i.e. builtin foo.bar might not support some arg or kwarg, so we can replace it at runtime with one that does.

@robert-hh
Copy link
Contributor

MicroPython is not CPython. I do not like the idea of changing names for long established classes, methods, functions. That would by a big pain for people actually using micropython for their projects and might want to update the firmware e.g. for bug fixes.

@jimmo
Copy link
Member Author

jimmo commented Aug 5, 2022

MicroPython is not CPython. I do not like the idea of changing names for long established classes, methods, functions. That would by a big pain for people actually using micropython for their projects and might want to update the firmware e.g. for bug fixes.

@robert-hh Just to clarify, I'm not proposing changing the names of any classes, methods, or functions. Only modules.

In terms of backwards compatibility it would be very easy to continue to provide the existing weak links / aliases for ufoo->foo (except in the reverse direction to the current implementation, such that the module would now be called foo). Additionally, of course if we moved e.g. os.mount to storage.mount then we would continue to alias it in os.

The main breaking change I'm proposing would be that builtins always take precedence and are never searched on the filesystem.

@mattytrentini
Copy link
Contributor

Thanks for the really clear, well thought-through write-up @jimmo!

+1 to remove the ufoo/weak link model. I've found it to be one of the most confusing topics for beginner/intermediate MicroPython users - and one of the more difficult to explain.

Adding extra kwargs to existing methods -- maybe ok?

I'd prefer not - but I can imagine some compelling cases that could be convincing.

Should we find new homes for existing non-CPython extensions...how long should we keep aliases...

Yes to finding new homes. Not long would be my preference - make painful changes quickly. More important to me: Making breaking changes like this - and their workarounds - very clear in release notes.

@dlech
Copy link
Contributor

dlech commented Aug 5, 2022

and the other historical reasons for the "u" prefix aren't compelling either.

A very compelling use case for me is that the "u" prefix allows us to use existing Python intellesene tools in IDEs to get correct code completion and type hints for the MicroPython version of a module. Existing tools don't have a way of saying "I want to replace the Python standard library with my own .pyi files". So being able to import ufoo sidesteps this and allows one to very easily benefit from coding tools without any extra work (other than creating the ufoo.pyi files for the MicroPython API).

In terms of backwards compatibility it would be very easy to continue to provide the existing weak links / aliases for ufoo->foo

As long as this is possible, then the proposed changes aren't really breaking then are they (other than the aliases may be disabled by default instead of enabled)?

@tannewt
Copy link

tannewt commented Aug 5, 2022

  1. It's worked so far for CircuitPython, and it's entirely possible that many MicroPython users have never needed/wanted it either (I know this isn't quite true).

Here is my perspective and what I use to guide CircuitPython.

  1. Having the builtin modules without the u prefix is almost always what people use because they have some background from CPython. If they don't, then teaching the non-u version is still more portable knowledge.
  2. CircuitPython code should run in CPython without changes. CPython code running in CircuitPython is not a goal because CPython written code tends to treat RAM as effectively infinite.

So, for 1, we've dropped the utime prefixes and for 2 we've dropped or moved all non-CPython APIs to other modules like storage or multiterminal. The CPython-compatible versions (builtins and libraries) shouldn't have anything added that will cause an error in CPython (extra kwargs would for some functions I believe.). We want imports to represent what people are using and cause errors early on start up rather than some unknown time later at first use. Moving extra functionality to other modules also makes it easy to provide that functionality in CPython with a library (see Blinka.)

Regarding allowing extending builtins, do you really need it? Why can't the calling code be changed to use a different module name or simply import foo_ext as foo? foo_ext can still import foo to use the builtin version.

I also have run into u* libraries that may have intended to match their CPython equivalent but then varied. I think the u* naming in libraries is a bit of a scapegoat to be similar instead of a strict subset. I find that variation confusing because it doesn't match one's expectations. Leaving the u* naming behind will lead to more different names for different modules and I think that'd be good.

@andrewleech
Copy link
Contributor

andrewleech commented Aug 5, 2022

I'm 100% in favour of this. Extending builtins is a surprisingly common practice in even cpython (in my experience), and even more so in micropython. As @jimmo mentioned this aids in keeping the built-in modules lean while allowing users to extend as needed for their application.

The u prefix is confusing for a lot of new users I encounter though.

It's true that the both the current mechanism and the proposed change comes with a ram cost in the duplication of the module table with from ufoo import *, though this could be replaced with amod.getattr function pattern similar to uasyncio to import on demand.

Similarly, the new system could be implemented in C with the built-in getattr/setattr functions; make mod.setattr create a __dict__ if needed and store the added features there, with getattr checking that. If __dict__ exists then it should get checked before the rom table else you wouldn't be able to override built-in functions.

Why can't the calling code be changed to use a different module name or simply import foo_ext as foo?

I've run into a number of cases where porting cpython code to micropython where the ufoo built-in is missing some needed features. Currently I can create a foo python module that provides these missing parts and not need to modify the module being ported. So I'm strongly against needing a _ext style renamed module to provide cpython compat. Often using this approach I've been able to use third party cpython code without any changes which is far better than needing to fork/patch/maintain just to support import changes.

@stinos
Copy link
Contributor

stinos commented Aug 6, 2022

Regarding allowing extending builtins, do you really need it?

Definitely yes, main reason being that most builtins are not 100% CPython-comaptible: the choice is then to either write a new module and provide functions, or to just do what is most convenient and user-friendly in all possible ways (code is CPython compatible, can just use standard CPython docs, no need to learn about another module and/or function name, ...)

@stinos
Copy link
Contributor

stinos commented Aug 6, 2022

to provide different extensions to the os module, and the user can choose which ones to install, and then "activate" them by importing them once at the top of main.py (or even boot.py)

For the unix-lik ports this would require some new mechanism (preferrably CPython-compatible, site.py perhaps) to have something run before everything else. At least I don't think we have that now. Possible (perf not measured) disadvantage is that in practice for most projects I have this would mean importing like 10 modules, always, even when none needed.

Option 1 is a no-go as far as I'm concerned, 3 only fixes the u prefix itself so not sure if it's worth it but for the rest it's interesting because it doesn't change a lot.

2 and 4 on the other hand are technically nice but have the issue mentioned above, and are not super convenient to write in general, and it also hurts discoverability somewhat (e.g. now if you want to know what's in os, you open os.py and/or moduos.c, not os_ext.py or so). Not sure which of 2, 4 and Andrew's approach with getattr would be 'best'. Would be interesting to see actual code as was shown for 4, just to be able to compare.

@jimmo
Copy link
Member Author

jimmo commented Aug 6, 2022

Thanks for the comments!

So being able to import ufoo sidesteps this and allows one to very easily benefit from coding tools without any extra work (other than creating the ufoo.pyi files for the MicroPython API).

@dlech OK interesting. This is something I hadn't considered, nor do I have any experience with.

My initial reaction is along the lines "surely there's got to be a better way" and to be honest we're already trying to make "use foo not ufoo" the default guidance, and so complicating that with "but ufoo fixes autocompletion" is unfortunate (and also means that you have to choose between supporting extensions-from-Python or autocompletion).

How bad is it that the autocomplete just uses Python's full completion? Is this a limitation of all IDE that they can't have project-aware auto-completion sources?

As long as this is possible, then the proposed changes aren't really breaking then are they (other than the aliases may be disabled by default instead of enabled)?

Yes, the only breaking change would be that we would stop the existing extension mechanism from working (i.e. builtins would now always take precedence).

Yes. Others will correct me if I'm wrong but I think our philosophy is much more: "MicroPython code can run in CPython with necessary shims, e.g. micropython.const, etc, and CPython fragments and occasionally whole modules should run unmodified.".

(@tannewt) Regarding allowing extending builtins, do you really need it? Why can't the calling code be changed to use a different module name or simply import foo_ext as foo?

(@andrewleech) I've run into a number of cases where porting cpython code to micropython where the ufoo built-in is missing some needed features.

(@tannewt) CircuitPython code should run in CPython without changes. CPython code running in CircuitPython is not a goal because CPython written code tends to treat RAM as effectively infinite.

@tannewt @andrewleech Yes, I guess this is exactly the crux of this conversation. This whole goal of extending built-in modules is only really necessary to make an existing file work completely unmodified.

In the "import foo_ext as foo" case, foo_ext still needs to do "from foo import *" (duplicate dict cost) or the module __getattr__ trick (extra code cost). It also doesn't work for more than one extension.

It's true that the both the current mechanism and the proposed change comes with a ram cost in the duplication of the module table with from ufoo import *, though this could be replaced with amod.getattr function pattern similar to uasyncio to import on demand.

This is a good idea, thanks @andrewleech . I think the whole "getattr + builtinimport hook" could be combined into a utility module too, so os_ext.py could look like:

import builtin_extension

# This gives us a __getattr__ forwarding to the existing os
# module (either built-in or a previous extesnion), and hooks
# "import os" for further usage to return this module instead.
builtin_extension.apply("os", __name__)

def foo():
  pass

Here is an implementation of builtin_extension.py to support this -- https://gist.github.com/jimmo/b918b73abc2f5c107b102d2ddb7a7976.

Similarly, the new system could be implemented in C with the built-in getattr/setattr functions; make mod.setattr create a dict if needed and store the added features there, with getattr checking that. If dict exists then it should get checked before the rom table else you wouldn't be able to override built-in functions.

Yes, exactly. This is along the lines of how the existing sys and builtin overrides work. Some consideration to how to do this with the absolute minimum RAM overhead (i.e. you need a pointer for that dict, and you don't want to do that for ~30-40 modules). There are ways to solve this though.

Definitely yes, main reason being that most builtins are not 100% CPython-comaptible: the choice is then to either write a new module and provide functions, or to just do what is most convenient and user-friendly in all possible ways (code is CPython compatible, can just use standard CPython docs, no need to learn about another module and/or function name, ...)

@stinos Does Scott's point above with "import foo_ext as foo" work for you? Or, like Andrew, do you want the CPython-compatible file to run exactly as-is without any modifications to import.

For the unix-lik ports this would require some new mechanism (preferrably CPython-compatible, site.py perhaps) to have something run before everything else. At least I don't think we have that now.

@stinos I'm not quite sure I follow.

I had imagined that the top of main.py (or for Unix/Windows, whatever your entry point is) would do this.

Possible (perf not measured) disadvantage is that in practice for most projects I have this would mean importing like 10 modules, always, even when none needed.

You could also imagine that some sort of automatically maintained script that imported all your installed extension packages could be useful. But conceptually I see this more as "I'm explicitly enabling the extra functionality I need for my app".

it also hurts discoverability somewhat (e.g. now if you want to know what's in os, you open os.py and/or moduos.c, not os_ext.py or so).

Do any of the approaches solve this? If we're going to make it possible to extend a module, it needs to be implemented across multiple files.

Not sure which of 2, 4 and Andrew's approach with getattr would be 'best'. Would be interesting to see actual code as was shown for 4, just to be able to compare.

Which one do you want to see code for? The implementation of mutable builtin-modules for 2?

@stinos
Copy link
Contributor

stinos commented Aug 6, 2022

Or, like Andrew, do you want the CPython-compatible file to run exactly as-is without any modifications to import.

Yes I'd really prefer this to stay as it is now, such that we can write code which is CPython compatible with the least friction, i.e. just import os or os.path etc (just to name the two probably used most examples of files which we have extensions to builtin functionality for).

edit just to give an idea, there are 43 matches for import os in our main codebase; so in principle that's just a find/replace to change that into import os_ext as os but that's just a step back, and a lot of friction

I had imagined that the top of main.py (or for Unix/Windows, whatever your entry point is) would do this.

When prototyping, writing (unit)tests etc I just want to create a new file and start writing 'normal' code, then run it (1). I wouldn't want to have to manually add import foo_ext to every single file which needs builtin foo + extension. Meaning instead I'd want whatever command (1) is to do that automatically. Which eventually boils down to at interpreter startup importing a custom module automatically which in turn has all the import xxx_ext statements. There are a couple of ways to do that, but perhaps makes sense to provide something for the unix port in the mainline.

Do any of the approaches solve this? If we're going to make it possible to extend a module, it needs to be implemented across multiple files.

No, but it's a minor issue. Also because as long as the extension is written in Python most text editor's 'go to symbol' etc will find it anyway.

Which one do you want to see code for? The implementation of mutable builtin-modules for 2?

Yes, to see how it compares in practice with the approach for 4

@dlech
Copy link
Contributor

dlech commented Aug 6, 2022

How bad is it that the autocomplete just uses Python's full completion?

You end up using modules/classes/methods/args that don't exist in MicroPython and your code fails at run time. Then you have to go find the relevant MicroPython documentation to figure out what is going on. This slows down development and is confusing for inexperienced programmers.

Is this a limitation of all IDE that they can't have project-aware auto-completion sources?

Generally, the way these tools work is that you say I want to use X Python runtime. This is specified by the absolute path to python executable and will use that find all site packages, etc this way. Usually this a virtual environment where you have all of the dependencies for your project installed. But since MicroPython is not fully CPython compatible, you can't just plug in the path to MicroPython here or create a MicroPython virtual environment.

@andrewleech
Copy link
Contributor

andrewleech commented Aug 6, 2022

I had imagined that the top of main.py (or for Unix/Windows, whatever your entry point is) would do this.

When prototyping, writing (unit)tests etc I just want to create a new file and start writing 'normal' code, then run it (1). I wouldn't want to have to manually add import foo_ext to every single file which needs builtin foo + extension.

I concur, currently if you have all the libraries "upip installed" or equivalent then just running files have all the libs on the path ready to import. The _ext scheme would not work this way so needs a pre-process step to get these on the path.
Cpython has the concept of .pth files that can be in a virtualenv that get run at the start of every python startup that allow python code to setup the env. On boards, the boot.py can also to this. On Unix etc I think it works be good to find/run a boot.py (or similar) automatically at startup if it exists on path.

You end up using modules/classes/methods/args that don't exist in MicroPython and your code fails at run time. Then you have to go find the relevant MicroPython documentation to figure out what is going on. This slows down development and is confusing for inexperienced programmers.

Is there a particular ide / autocomplete package you're using currently? Autocomplete and code navigation is something I use daily myself but have just gotten used to the sub-par behaviour of cpython stubs. This certainly isn't ideal though.

A proper micropython autocomplete setup is something I'm keen to get working. I know there are some stubbing tools it there that should do most of the work for us, perhaps arranging the outputs of that into something an ide could recognise as a python env would be possible? Maybe that's something that could be built on top of the new manifest system to include copies/links to libraries installed as well.

@dlech
Copy link
Contributor

dlech commented Aug 6, 2022

Is there a particular ide / autocomplete package you're using currently?

We're using Pylance(based on Pyright) in VSCode and also Jedi.

A proper micropython autocomplete setup is something I'm keen to get working.

Microbit forked Pyright to make it work in the browser and with their MicroPython API. You can see it in action at https://python.microbit.org/v/beta. Making this work in general for MicroPython with some sort of manifest system as you have describe would be ideal IMHO.

@jimmo
Copy link
Member Author

jimmo commented Aug 19, 2022

Some notes on the performance / memory...

By removing the filesystem search from importing a builtin, import os (or import uos) goes from ~2-5ms to 0.15ms. (Note: in current builds import uos should actually be skipping the filesystem, but there's a bug where it still queries the path in case there's a next level). Note the 2-5ms is a best case with the default sys.path and a tiny filesystem, and it scales linearly with the number of imports (i.e. there's no caching effect).

With option 4 (i.e. hooking __import__) using builtin_extension.py linked above.

It takes 3us to do os.uname with the "real" os module (this is just the cost of the .uname part). When os is extended, accessing the new methods (e.g. os.foo) takes 3us, but inherited methods (e.g. os.uname) now take 20us (this is the overhead of executing the Python code for __getattr__. And then it's +17ms for every additional extension you add.

I'm not sure if this is a big concern... if this was performance critical, then os.xyz should be cached in a local variable to avoid the lookup anyway. There are some easy optimisations though, for example if we add a mechanism for accessing mp_obj_new_bound_meth from Python (i.e. equivalent to a one-argument version of functools.partial), then you can get this down to 9us.

sys.modules[ext_name].__getattr__ = micropython.bind(getattr, mod)

Even simpler if you just want to make a particular method fast you can write os.foo = os.foo to add it to the globals dict of the "outer" module.

As far as memory overhead, installing the extension costs 224 bytes of RAM for the import hook etc.

@laurensvalk
Copy link
Contributor

We use the "u" prefix for built-in modules for several reasons (..). The primary reason though is to allow foo.py to exist on the filesystem and extend the built-in ufoo. (...)

I've always thought of the u-modules as accelerator modules, only using u instead of _. From PEP 399:

The Python standard library under CPython contains various instances of modules implemented in both pure Python and C (either entirely or partially) (...) If an acceleration module is provided it is to be named the same as the module it is accelerating with an underscore attached as a prefix, e.g., _warnings for warnings. The common pattern to access the accelerated code from the pure Python implementation is to import it with an import *, e.g., from _warnings import *.

I'm not arguing to do this exactly, but it seems like there is some precedent that could provide inspiration here.

It's weird that when you write import foo; print(foo) you get ufoo. Also help('modules') lists everything as ufoo.

It seems like implementation could address this, but having the accelerator modules shining through isn't inherently wrong/weird. For example in Python3, print(sys.stdin) gives <_io.TextIOWrapper ...

This is a CPython incompatibility (in CPython you cannot replace a built-in from Python via the filesystem..

But you can extend an accelerator module _foo as foo.

It's just generally confusing and difficult to document and explain. We are slowly improving this, but there's still a lot of code out there writing "import ufoo".

With this pattern in mind, the end user and documentation will always use import foo. The ufoo module (think of it as _foo) would remain available for library builders.

The difference from PEP 399 is that MicroPython wouldn't actually always use a pure python wrapper to make this work, but this comes down to implementation. The existing weak links approach is just MicroPython's current way of doing it, but this could perhaps be improved.


So I guess my point is that having u-modules in the implementation makes sense and has some CPython precedent.

As mentioned above, end-users should be able to import foo and the documentation just needs to have foo. It is certainly an (ongoing) effort to make it work like users expect, but I don't see this is an argument against having the u-modules in the implementation.

End-users (especially beginners) would never have to see or use the u-modules, but library builders can still use them.

@jimmo
Copy link
Member Author

jimmo commented Aug 19, 2022

Thanks @laurensvalk

... PEP399...
I'm not arguing to do this exactly, but it seems like there is some precedent that could provide inspiration here.

Yes, this is exactly how _uasyncio provides an "accelerated" Task etc. Although our implementation is a bit different because we just don't include task.py at all when _uasyncio is available.

It's also relevant to #8968 where one of the options being considered is to freeze ssl.py and provide _ssl as the low-level implementation. (But in this case it's conceptually a bit different because there is no pure-Python version).

It's a bit difficult to apply this as a precedent to this issue though because you still can't override os in CPython. And the way accelerator modules are used are definitely not at all "micro" in its implementation (i.e. the whole Python implementation gets loaded only to be replaced in its entirety with the accelerator module).

So I guess my point is that having u-modules in the implementation makes sense and has some CPython precedent.
As mentioned above, end-users should be able to import foo and the documentation just needs to have foo. It is certainly an (ongoing) effort to make it work like users expect, but I don't see this is an argument against having the u-modules in the implementation.

My goal here is that I don't think our umodule/weak link system is actually the best way to provide the functionality that users want/need.

  • Many (most?) users don't need to extend builtins. So we should make that fast and efficient. (e.g. avoid the ~2-5ms import cost of searching the filesystem for a module that isn't there for each built-in module).
  • Some users need a small number of extra functions that aren't provided by our built-ins. As @tannewt said, in many cases this doesn't even need to be made available as a built-in extension, i.e. they need the method, how it gets imported isn't such a big deal.
  • Some users want to use a CPython library verbatim. In that case having some mechanism to extend a built-in with missing functionality is required to support this use case.
  • We want to provide the "missing functions" in micropython-lib (and third party packages too), but at the same time it's wasteful to write large packages that provide 10 extra methods when the user just wanted one thing, rather it would be more efficient to write fine-grained extension packages. We also might want to combine extensions from multiple sources. So we need a way to compose extensions (this isn't currently possible with ufoo/weak links).
  • Some "built-in" things that require extending aren't built-ins (e.g. frozen modules like uasyncio). It would be really good to have a single standard way to extend any library with whatever is needed for a given app.
  • Using from ufoo import * is wasteful as an implementation of a built-in. (We could recommend the __getattr__ pattern as an alternative though)

but library builders can still use them.

I'd be interested to know more about your specific use case with pybricks. Do you provide pre-built firmware with frozen in extensions? Or do your users add them to the filesystem? Are they writing their own or using micropython-lib ones (or other sources?). Would any of the options outlined in this issue work for you & your users?

@laurensvalk
Copy link
Contributor

Thanks for your response. That led me to a subtlety that may be worth clarifying --- Reading back through this thread, it seems that not all comments about extending builtins are about the same thing.

If I understand the posts in favor of extending builtins correctly, most would like to extend uos, not necessarily extend os.

But both variants are implicitly or explicitly discussed in this thread, and I'm not sure the replies here always respond to the same thing.

I'll follow up with my personal opinion in the next post.

@laurensvalk
Copy link
Contributor

So with that in mind, I fully agree with the motivation in @tannewt's post, but with a subtle clarification of the implication --- moving away from the uos name could be fine, so long as it doesn't become os. I believe it would be better to reserve os for something that actually behaves like os, however it gets implemented.

I don't think our umodule/weak link system is actually the best way to provide the functionality that users want/need.

Agreed. My comments so far are mainly about keeping the internal implementation modules (e.g. uos) available under a distinct name, so that os remains reserved something closer to the real deal.

Some notes on the performance / memory

I understand the concerns on RAM and build size, but how important is import time?

I'd be interested to know more about your specific use case with pybricks. Do you provide pre-built firmware with frozen in extensions? Or do your users add them to the filesystem? Are they writing their own or using micropython-lib ones (or other sources?). Would any of the options outlined in this issue work for you & your users?

Most of this doesn't affect Pybricks directly, at least not right now. I wrote this from a generic point of view, and with the backdrop of having written several coding books for kids.

Several posts have mentioned that some solutions are easy to explain. I believe it's even better when it's easy to understand. So if two things work differently, even subtely, then in my view they shouldn't have the same name.

I understand that this doensn't resolve the question at hand, but I'd have to read up a bit more on existing efforts before commenting on any of the proposed solutions.

@dpgeorge
Copy link
Member

Several posts have mentioned that some solutions are easy to explain. I believe it's even better when it's easy to understand.

I think the best/easiest case is when there is nothing there at all, when there's nothing to explain. And removing u-naming is a way to simplify things such that there's no longer any difference to CPython and hence nothing to document/teach.

Of course that's not the whole story because we still need a way to override built-in modules, so there will be something extra and something to document/teach. But for most use cases / most users / 80% of the time / to first order, things should just behave like CPython without having to worry about differences. import os should just work. On top of that, for the other 20%, to extend os, you need to learn something.

At the moment it feels like it's the other way around, you need to first learn about u-naming before you can do anything.

@iabdalkader
Copy link
Contributor

Sorry if I've missed this in the discussion, but did you consider the case where a frozen module adds bits/extends a built-in module ? For example if foo module in micropython-lib with from ufoo import *, is frozen via manifest.py, this as far as I know currently works, but how would it work after this change ? Can options 3/4 detect frozen vs built-in and give precedence to frozen modules ?

@tannewt
Copy link

tannewt commented Oct 11, 2022 via email

@Josverl
Copy link
Contributor

Josverl commented Oct 11, 2022

I maintain similar stubs for MicroPython. Most of them are generated automatically from the repo, and try to consider variations across, ports and boards.
Pyright has been adopted to be able to use these as there were some issues with overriding some of the stdlib modules.
Once settled on a model it should be straightforward to adjust the stubs accordingly and publish them, install the stubs in a folder or venv , and the tools should pick them up from there.

@jimmo
Copy link
Member Author

jimmo commented May 10, 2023

See #11456 which opens another option for extending built-ins from Python.

@Gadgetoid
Copy link
Contributor

#11456 brought me here: I had no idea what "u"modules were for and have been using them interchangeably and passing on that code smell to whoever uses/learns-from our examples. (Granted, my existence in mostly the make-everything-a-C-module space of MicroPython does not lend itself well to finding and understanding these quirks, which is why I'm lurking the GitHub and trying to expand my knowledge.)

It's pretty easy to see the extent of this across our MicroPython examples: https://github.com/search?q=repo%3Apimoroni%2Fpimoroni-pico+%22import+u%22&type=code

Granted not all of these are a problem. But it does make uasyncio and urequests and umqtt rather confusing. Are those a cosmetic prefix or an intentional disambiguation from similarly-named CPython modules?

In our case, I guess I should raise an issue/PR to switch from uos to os, ujson to json and so on.

@jimmo
Copy link
Member Author

jimmo commented May 10, 2023

#11456 brought me here: I had no idea what "u"modules were for and have been using them interchangeably and passing on that code smell to whoever uses/learns-from our examples.

It's pretty easy to see the extent of this across our MicroPython examples: https://github.com/search?q=repo%3Apimoroni%2Fpimoroni-pico+%22import+u%22&type=code

I think Damien's comment #9018 (comment) is about exactly this point -- we should just make it simple and obvious.

Granted not all of these are a problem. But it does make uasyncio and urequests and umqtt rather confusing. Are those a cosmetic prefix or an intentional disambiguation from similarly-named CPython modules?

There's an additional piece of historical context (and a third function of the u-prefix) there which was that it disambiguated the package name on PyPI. This is no longer an issue.

There is an open issue to rename requests. See micropython/micropython-lib#540
The main decision there is the best way to provide backwards compatibility, but I think we have decent options and we should do that at the same time as the built-in module rename discussed here.

We should do the same thing for asyncio. (This one is super confusing because it's a built-in in behavior, i.e. it's frozen, but it's not actually a built-in so doesn't get the "weak links" treatment).

In our case, I guess I should raise an issue/PR to switch from uos to os, ujson to json and so on.

Yep, see the official guidance here:
https://docs.micropython.org/en/latest/library/index.html#extending-built-in-libraries-from-python

Specifically: "Other than when you specifically want to force the use of the built-in module, we recommend always using import module rather than import umodule."

@jimmo
Copy link
Member Author

jimmo commented Jun 2, 2023

See #9069 for an implementation of this.

In the end I went with a fifth option which doesn't change any behavior but still allows removes the u-prefix from the module objects.

  1. For now import umodule continues to work, but the preferred mechanism to force a built-in import is to temporarily clear sys.path as described and implemented in py/builtinimport: Allow built-in modules to be packages (v3) #11456. There is scope to reduce the number of allocations (see py/builtinimport: Allow built-in modules to be packages (v3) #11456 (comment)).

This means that there should be no user-visible change other the output of help("modules") and what you see if you print a module object (i.e. <module 'os'> rather than <module 'uos'>).

Later we can consider removing the handling for umodule (or at least making it an mpconfig option).

@jimmo
Copy link
Member Author

jimmo commented Jul 4, 2023

This was implemented in #9069 and #11740:

  • All built-in modules (and asyncio) were renamed to remove the u-prefix.
  • There's a new mechanism for forcing a built-in based on sys.path.
  • import umodule still also works as a backwards-compatible way of forcing a built-in.

@jimmo jimmo closed this as completed Jul 4, 2023
smurfix added a commit to M-o-a-T/micropython that referenced this issue Oct 22, 2023
U-module renaming, deflate module, IDF 5, board variants and Pico-W BLE

This release of MicroPython sees the renaming of built-in modules to remove
the u-prefix, a new deflate module with optional compression support, the
introduction of board variants, switching of the esp32 port to use IDF 5
together with improved heap management, support for BLE on RPi Pico W
boards, and STM32H5xx support.  The project is also now using codespell and
ruff to improve code quality.  New boards added in this release are:
ARDUINO_NANO_ESP32 and UM_NANOS3 (esp32 port), ADAFRUIT_METRO_M7 (mimxrt
port), ARDUINO_PORTENTA_C33 and VK_RA6M5 (renesas-ra port),
ADAFRUIT_METRO_M4_EXPRESS (samd port), NUCLEO_L4A6ZG and STM32H573I_DK
(stm32 port).

The renaming of built-in modules to remove the u-prefix -- for example
utime becomes time, uasyncio becomes asyncio -- is done to improve
compatibility with CPython and eliminate confusion about whether to import
the u-version or the non-u-version of the name.  Now, one should just
always import the non-u-version and no longer think about u-naming.  The
only remaining module with a u-prefix is uctypes because it is not
compatible with the CPython ctypes modules.  The following modules are
still available via their u-names for backwards compatibility: array,
asyncio, binascii, bluetooth, collections, cryptolib, errno, hashlib,
heapq, io, json, machine, os, platform, random, re, select, socket, ssl,
struct, time, websocket.  These modules (except for asyncio) are also
extensible and can be overridden by a file with the same name, eg time.py.
To force the import of a built-in, one must first clear (and subsequently
restore) sys.path; the ability to write to the sys.path attribute has also
been implemented in this release.  For further information see the
discussion at micropython#9018.

Furthermore, importing itself has been tweaked and optimised, and importing
of built-ins no longer touch the filesystem, which makes a typical built-in
import take ~0.15ms rather than 3-5ms.  For modules that fail to import,
they are now removed from sys.modules, allowing the import to be attempted
again.  This required adding "NLR jump callbacks" to efficiently run some
code if an exception is raised.

Additional improvements to the core runtime include support for conversion
specifiers in f-strings (e.g. !r), speeding up of incremental GC cycles by
tracking the last used block, addition of a new MICROPY_GC_SPLIT_HEAP_AUTO
"auto grow heap" mode and support for pad bytes in struct format.  The
documentation, examples and test have also seen general improvements and
additions.

For the extended modules, the zlib C module has been removed and replaced
with a new MicroPython-specific deflate module and DeflateIO class that is
optimised to provide efficient streaming compression and decompression.
The zlib (and gzip) modules are now implemented in pure Python on top of
the deflate module.  The timeq module has been removed, it existed only for
a previous version of (u)asyncio and is no longer used.  In the ssl
module, SSLContext has been added to be more compatible with CPython.  The
select module now supports using system/POSIX poll when possible and the
unix port now uses this implementation instead of its own one.  That means
the unix port can now select/poll on custom Python objects that implement
the appropriate ioctl.  The socket module now supports SO_BROADCAST, and
sys.std*.buffer objects now exist on unix.  There is also a new esp-hosted
network driver for external ESP32-based network coprocessors.

mpy-cross now allows reading source files from stdin and writing compiled
code to stdout.

The esp8266 and esp32 ports add support for the Espressif ESP-NOW protocol,
and the ability to set/get the power saving mode of the WLAN hardware.  The
esp8266 port adds board-variant support, combining all boards into a single
ESP8266_GENERIC with FLASH_512K, FLASH_1M and OTA variants.

The esp32 port has now switched exclusively to ESP-IDF 5, and all existing
components have been updated to work with this new IDF (except
esp32.hall_sensor() which has been removed).  The FSM ULP has been enabled
for S2 and S3 chips, sockets now support SO_BINDTODEVICE, and board-named
pins and the Pin.board dict have been implemented.  The MicroPython heap
has been reworked on this port to support the large variety of memory
configurations: it now starts at 64kbytes and automatically grows as
needed, with new segments allocated from the IDF heap.  This means that
boards with SPIRAM have much faster GC collection times if only a small
amount of RAM is used, all available RAM can be used if needed, and the IDF
has access to enough RAM for things like SSL sockets.  For more information
see micropython#12141.  The esp32 port
has also had its generic boards consolidated and renamed to ESP32_GENERIC,
ESP32_GENERIC_C3, ESP32_GENERIC_S2 and ESP32_GENERIC_S3, and some now have
variants such as SPIRAM.

The mimxrt port has fixes and improvements to PWM and Pin.irq, as well as
support for UART hardware flow control and UART.deinit.  It has also seen
integration of WiFi via the CYW43 driver, and Bluetooth via NimBLE.  The
mbedTLS bindings have enabled time validation of certificates.

The renesas-ra port has changed board names to match the product name,
updated to use FSP v4.4.0, added support for: RA6M5 MCUs, machine PWM, DAC,
RNG and SDCard classes, TinyUSB, Bluetooth via NimBLE, networking via lwIP,
and mbedTLS.

Bluetooth support has also been added to the rp2 port for the Pico W board.
And the RPi boards have been renamed from PICO to RPI_PICO, and PICO_W to
RPI_PICO_W.  Lightsleep has been fixed on this port so it works while WiFi
is powered off, and time.time_ns() now has microsecond resolution.

The samd port sees the addition of SPI and QSPI flash drivers to support
filesystems on external flash.

The stm32 port add support for STM32H5xx MCUs, basic support for the
OCTOSPI peripheral, and USB support for STM32L1xx MCUs.  New functions have
been added to the stm module to support the sub-GHz radio on STM32WL55.

In micropython-lib, an extensive LoRa module has been added along with
drivers for SX126x and SX127x chipsets, and the STM32WL55.  This module
supports both synchronous and asynchronous (asyncio) mode.  Also, as part
of the u-module renaming, urequests has been renamed to requests (but for
backwards compatibility "import urequests" still works for now).

The change in code size since the previous release for various ports is
(absolute and percentage change in the text section):

       bare-arm:    +192  +0.340%
    minimal x86:    +310  +0.169%
       unix x64:   +4784  +0.610%
          stm32:    -524  -0.134%
         cc3200:    +280  +0.154%
        esp8266:   +8016  +1.151%
          esp32: +112133  +7.293%
         mimxrt:   +3624  +1.015%
     renesas-ra:   -2184  -0.348%
            nrf:    +616  +0.332%
            rp2:   +1920  +0.595%
           samd:   -7904  -2.953%

The changes that dominate these numbers are:
- bare-arm, minimal, cc3200, nrf: NLR jump callbacks and support for
  extensible modules
- unix: update of mbedTLS to v2.28.1, support for polling Python objects in
  select module
- stm32: removal of the timeq module
- esp8266: addition of the espnow module
- esp32: switching to ESP-IDF 5
- mimxrt: UART features, time validation of SSL certificates
- renesas-ra: move to FSP v4.4.0, remove timeq module
- rp2: machine.PWM enhancements
- samd: drop support for VfsLfs1

Performance is effectively unchanged since the previous release.

Note that this is the last release to use the current versioning scheme for
nightly/unstable builds, whereby a build between releases is versioned as
v1.20.0-<num>-g<hash> (following the release of v1.20.0).  Moving forward,
nightly builds will now be called preview builds and be versioned with the
next release number.  For example, if the last release was v1.21.0 then
preview releases will be of the form v1.22.0-preview.<num>.g<hash>.  For
discussion see micropython#12127.

Thanks to everyone who contributed to this release:
Adam Green, Alexander Wilde, algonell, Andrew Leech, Andy Piper, Angus
Gratton, Armin Brauns, brave ulysses, Brett Cannon, Brian 'redbeard'
Harrington, Carlosgg, Chris Wilson, Christian Clauss, Damien George, Damien
Tournoud, Daniël van de Giessen, David Grayson, David Lechner, David Yang,
dotnfc, Duncan Lowther, Elecia White, elibdev, Elvis Pfutzenreuter, Felix
Dörre, Francis Dela Cruz, Glenn Moloney, glenn20, iabdalkader, Ihor
Nehrutsa, Jared Hancock, Jim Lipsey, Jim Mussared, Jon Nordby, Jonas
Scharpf, Jos Verlinde, Kwabena W. Agyeman, Luca Burelli, marble, Mark
Grosen, mbedNoobNinja, mcskatkat, Mingjie Shen, Mirko Vogt, Nicholas H.
Tollervey, Oliver Joos, Ondrej Wisniewski, patrick, Peter Harper, Phil
Howard, Philipp Ebensberger, Rene Straub, robert-hh, Sebastian Romero, Seon
Rozenblum, stephanelsmith, stijn, Takeo Takahashi, Thomas, Tobias
Thyrrestrup, UnexpectedMaker, Victor Rajewski, vsfos, Wang Xuancong, Wanlin
Wang, Wilko Nienhaus, Wind-stormger, Yaroslav Halchenko, Yilin Sun, Yuuki
NAGAO.

The work done in this release was funded in part through GitHub Sponsors,
and in part by George Robotics, Planet Innovation, Espressif, Arduino, LEGO
Education and OpenMV.

What follows is a detailed list of changes, generated from the git commit
history, and organised into sections.

Main components
===============

all:
- fix spelling mistakes based on codespell check
- fix strings with backslash by using raw string literals
- fix various Python coding inconsistencies found by ruff
- fix cases of Python variable assigned but never used
- rename MP_QSTR_umodule to MP_QSTR_module everywhere
- rename mp_umodule*, mp_module_umodule* to remove the "u" prefix
- rename mod_umodule*, ^umodule* to remove the "u" prefix
- rename UMODULE to MODULE in preprocessor/Makefile vars
- rename *umodule*.h to remove the "u" prefix
- rename *umodule*.c to remove the "u" prefix
- use MP_REGISTER_EXTENSIBLE_MODULE for overrideable built-ins
- replace all uses of umodule in Python code
- remove the zlib module
- remove query-variants make target
- add missing imports for micropython.const
- add Black configuration section to pyproject.toml
- add ruff to pre-commit
- CODECONVENTIONS: require that commits be signed-off by the author

py core:
- ringbuf: implement put_bytes/get_bytes functions
- parse: fix build when COMP_CONST_FOLDING=0 and COMP_MODULE_CONST=1
- compile: remove over-eager optimisation of tuples as if condition
- stackctrl: add gcc pragmas to ignore dangling-pointer warning
- gc: make improvements to MICROPY_GC_HOOK_LOOP
- obj: remove mp_generic_unary_op()
- objslice: ensure slice is not hashable
- objdict: fix __hash__ for dict_view types
- objarray: disallow memoryview addition
- objstr: return unsupported binop instead of raising TypeError
- runtime: if inplace binop fails then try corresponding normal binop
- change MP_UNARY_OP_INT to MP_UNARY_OP_INT_MAYBE
- obj: accept user types in mp_obj_get_int_maybe
- objint: allow int() to parse anything with the buffer protocol
- builtinimport: handle empty sys.path correctly
- builtinimport: optimise sub-package loading
- builtinimport: allow builtin modules to be packages
- objmodule: don't use sys.modules to track a builtin __init__
- nlrsetjmp: use MP_NLR_JUMP_HEAD macro to simplify code
- nlr: remove commented-out debugging code
- nlr: implement jump callbacks
- use nlr jump callbacks to optimise compile/execute functions
- builtinimport: remove partially-loaded modules from sys.modules
- builtinimport: remove weak links
- makemoduledefs.py: add a way to register extensible built-in modules
- objmodule: add a table of built-in modules with delegation
- objmodule: workaround for MSVC with no module delegation
- mpconfig: enable module delegation if sys needs it
- modsys: allow sys.path to be assigned to
- mkrules.mk: automatically configure frozen options when manifest set
- parsenum: fix typo in #endif comment
- nlraarch64: fix dangerous use of input register
- makemoduledefs.py: fix declaring multiple module delegations
- makemoduledefs.py: automatically declare delegation attr functions
- lexer: allow conversion specifiers in f-strings (e.g. !r)
- mkrules.mk: allow $(AFLAGS) to set flags to $(AS)
- compile: fix async for's stack handling of iterator expression
- builtinimport: fix built-in imports when external import is disabled
- stream: add mp_stream___exit___obj that calls mp_stream_close
- runtime: always initialise sched_state in mp_init
- mpconfig: add MICROPY_PY_PLATFORM, enabled at extra features level
- gc: speed up incremental GC cycles by tracking the last used block
- gc: apply some code formatting cleanup
- gc: add new MICROPY_GC_SPLIT_HEAP_AUTO "auto grow heap" mode
- profile: remove the requirement to disable MICROPY_COMP_CONST
- mpconfig: enable SSL finalizers if finalizers are enabled
- objstr: fix `str % {}` edge case
- modstruct: support pad bytes in struct format
- dynruntime.h: implement MP_OBJ_NEW_QSTR
- modthread: return thread id from start_new_thread()
- malloc: fix DEBUG_print() args in m_realloc_maybe
- runtime: add helpers to call a general function on nlr jump callback
- parse: always free lexer even if an exception is raised
- persistentcode: always close reader even if an exception is raised
- gc: add "max new split" value in result of gc.mem_free()
- nlrx64: mark nlr_push() as naked function when possible
- mkrules.mk: don't strip binary if STRIP variable is unset
- change ifdef DEBUG_PRINT to if DEBUG_PRINT
- lexer: add missing initialisation for fstring_args_idx

extmod:
- utime_mphal: provide a general mktime function
- modutime: provide a generic time module
- machine_pwm: remove PWM_INIT and PWM_DUTY_U16_NS config options
- network_cyw43: add power management constants
- vfs_lfsx: fix offset used before range check
- extmod.mk: suppress deprecated-non-prototype warning
- moduos: move os.sync() into extmod/moduos.c
- modtimeq: remove timeq module
- btstack: add cmake support for BTstack
- btstack: fix marking of static addresses in set_random_address
- asyncio: rename uasyncio to asyncio
- asyncio/uasyncio.py: add backwards-compatible uasyncio alias
- modbinascii: fix buffer length error
- update to support mbedtls 3.x
- modplatform: set MICROPY_PLATFORM_ARCH on riscv platforms
- modbtree: undefine queue macros before including berkeley-db
- modssl: add SSLContext class
- moddeflate: add deflate module providing the DeflateIO class
- modssl_mbedtls: reference SSLContext from SSLSocket
- vfs_posix_file: add poll support for missing ERR,HUP,NVAL values
- modselect: abstract out a poll_set_t struct and functions
- modselect: factor low-level polling code into common function
- modselect: add optimisation to use system poll when possible
- modselect: remove undocumented support for flags arg to poll
- modssl_mbedtls: reject ioctls that are not supported
- modssl_mbedtls: fix ioctl of a socket in closed/error state
- modselect: properly track number of poll objects that are fd's
- modssl_mbedtls: clear sock member if error creating SSLSocket
- moddeflate: change default window size
- vfs_posix_file: fix flush handling on macOS
- vfs_posix_file: implement sys.std*.buffer objects
- modlwip: add support for SO_BROADCAST socket option
- modsocket: add support for SO_BROADCAST socket option
- modssl_mbedtls: call func psa_crypto_init if PSA is used
- modssl_mbedtls: ignore err ERR_SSL_RECEIVED_NEW_SESSION_TICKET
- modlwip: fix setting of IP option SOF_BROADCAST
- network_esp_hosted: add ESP-Hosted networking interface
- modssl_axtls: only close underlying socket once if it was used
- asyncio/event.py: fix ThreadSafeFlag.ioctl return
- btstack/btstack_hci_uart: trigger a poll after UART data is sent
- asyncio/stream.py: fix cancellation handling of start_server
- modnetwork: increase max hostname length to 32
- modnetwork: forward if.config(hostname) to network.hostname
- vfs_posix_file: fix flush handling in msvc builds

shared:
- upytesthelper: fix spelling of "default"
- libc/printf: fix stdout destination for putchar and puts
- tinyusb: avoid symbol clash on targets with external TinyUSB
- tinyusb: support HS endpoint sizes
- netutils/dhcpserver: reply on correct netif

drivers:
- cyw43: make the CYW43 Bluetooth HCI driver more portable
- esp-hosted: add host driver for ESP-Hosted firmware
- ninaw10/nina_bt_hci: make some minor fixes to HCI driver
- esp-hosted: fix pin IRQ
- esp-hosted: fix MTU size
- esp-hosted: add support for WiFI LED activity indicator

mpy-cross:
- allow specifying source files starting with -
- allow reading from stdin and writing to stdout
- when reading from stdin, write output to stdout
- allow specifying stdin as input without --
- fix source file name in file-not-found error

lib:
- mbedtls_errors: update error list for current version of mbedtls
- mbedtls: update to mbedtls v2.28.1
- mbedtls_errors: add esp32-specific mbedtls error file
- mbedtls_errors: update patch and error list for new mbedtls
- mbedtls: update to mbedtls v2.28.3
- fsp: update FSP for renesas-ra to the latest version v4.4.0
- cyw43-driver: update driver to latest version v1.0.1
- btstack: update to v1.5.6.2
- pico-sdk: update to version 1.5.1
- stm32lib: update library for H5 v1.0.0
- oofatfs: fix speculative read in create_name
- uzlib: add memory-efficient, streaming LZ77 compression support
- uzlib/lz77: always use separate history buffer
- uzlib/defl_static: implement some code size improvements
- uzlib: clean up tinf -> uzlib rename
- uzlib: combine zlib/gzip header parsing to allow auto-detect
- uzlib/tinflate: implement more compact lookup tables
- uzlib/defl_static: optimize zlib_start/finish_block
- uzlib: add a source_read_data var to pass to source_read_cb
- tinyusb: update to the most recent master
- protobuf-c: add protobuf-c library
- cyw43-driver: update driver to latest version v1.0.2
- micropython-lib: update submodule to latest

Support components
==================

docs:
- reference: remove double 'are' in glossary
- update the PWM examples based on recent API improvements
- samd: make use of pin names more consistent in examples
- reference/mpyfiles: add release info on v6.1
- library/espnow: update espnow docs for WLAN.config(pm=x) options
- develop/porting: add missing code to example main.c and Makefile
- reference/speed_python: remove 4-arg limit for viper
- mimxrt: add the pin-out for the Adafruit Metro M7 board
- samd: add the pin-out for the Adafruit Metro M4 Airlift board
- library/index: update built-in extension docs
- reference/packages: add GitHub repo to package example dependency
- reference/mpremote.rst: extend the mpremote guide
- library/index: update docs after umodule rename
- rename uasyncio to asyncio
- esp32: update esp32 docs based on IDF v5 changes
- library/ssl: add documentation for SSLContext
- esp32/quickref: add LAN example for WT32-ETH01 version 1.4
- library/deflate: add docs for deflate.DeflateIO
- develop/gettingstarted: clarify submodule initialization
- develop/gettingstarted: update ARM package list
- library/neopixel: change link to a micropython-lib reference
- library/platform: add docs for the platform library
- library/network: clarify network.hostname() behaviour
- esp32/tutorial: add example for pin access via registers
- library/esp32: update ESP32 idf_heap_info docs to match behaviour
- library/gc: clarify mem_alloc and mem_free only for Python heap
- conf.py: add sphinxcontrib.jquery to extensions
- add requirements.txt file with dependencies for Sphinx
- change remaining "urequests" references to "requests"

examples:
- usercmodule: add a sub-package example
- natmod: rename umodule to module
- hwapi: rename uasyncio to asyncio
- natmod/deflate: add deflate as a dynamic native module
- mark asm, pio, etc. as noqa: F821 (undefined-name)
- hwapi: add missing import for 96Boards Carbon example
- bluetooth: raise ValueError when advertising data is too large
- bluetooth: link to aioble in BLE examples
- natmod: add features4 as a class definition example
- unix/machine_bios.py: fix typo

tests:
- run-tests.py: ensure correct cwd for mpy tests
- basics: add more tests for hashing of various types
- basics: remove __index__ and __inv__ from special methods tests
- import/builtin_ext.py: add test for built-in module override
- import/import_pkg9.py: add test for subpackage attribute
- replace umodule with module everywhere
- run-multitests.py: don't allow imports from the cwd
- run-perfbench.py: don't allow imports from the cwd
- run-natmodtests.py: don't allow imports from the cwd
- float: test domain errors for more combos of args to math funcs
- rename uasyncio to asyncio
- extmod/uctypes_array_assign_le: fix buffer
- extmod/framebuf: fix buffer size issues
- extmod: add tests for ssl.SSLContext
- extmod: add test for passing cadata into ssl.wrap_socket()
- extmod: add deflate.DeflateIO tests
- extmod: add coverage tests for select module
- extmod: skip select/socket tests if they can't create UDP socket
- extmod/select_poll_eintr.py: improve robustness of test
- misc/sys_settrace_features.py: fix to run on newer CPython
- unix/mod_os: make os.system() test work on windows
- run-tests.py: capture output of stderr when running on CPython
- multi_net: increase asyncio tests timeouts
- stress/bytecode_limit.py: reverse order of cases
- float/float_format_ints.py: put power-of-10 test in separate file
- extmod/deflate_decompress.py: skip test when not enough memory
- extmod/ssl_cadata.py: skip test on axtls
- float/math_domain.py: tweak test to also pass with obj-repr-C
- extmod/vfs_fat_finaliser.py: tweak test so files are collected
- README: document ./run-internalbench.py
- run-internalbench.py: remove old CPython reference
- multi_net/ssl_cert_rsa.py: update test certificate
- extmod/asyncio_threadsafeflag.py: update for unix select

tools:
- pyboard.py: rename ProcessPtyToTerminal member "ser" to "serial"
- mpremote: remove unused import of serial
- pyboard.py: import serial.tools.list_ports
- pyboard.py: import errno to fix undefined name in PyboardError
- manifestfile.py: fix license capturing
- mpremote: add repl option to escape non-printable characters
- pydfu.py: use getattr to retrieve getargspec function
- mpremote: add `sleep` command
- mpremote: allow terminator for shortcut commands
- mpremote: add `rtc` commands to get and set the RTC
- mpremote: handle `cp` without destination
- mpremote: detach mpremote from pyboard.py
- mpremote: fix use of stdout_write_bytes function
- mpremote: fix exec_ -> exec in commands.py
- autobuild: update auto-build code to build esp32 port with IDF v5
- autobuild: add support for application .bin files for esp32
- mpy-tool.py: use isinstance() for type checking
- codeformat.py: skip formatting ESP-IDF managed components
- codeformat.py: use pyproject.toml for black config
- mpremote: make soft-reset count as an action
- autobuild: automatically build all variants for each board
- mpy_ld.py: pre-declare some local variables to appease linter
- mpy-tool.py: ignore linter failure in Python 2 compatibility code
- mpy_ld.py: support more complex rodata sections
- metrics.py: fix nrf and rp2 board names after renaming
- autobuild: include .bin firmware in renesas-ra build output
- autobuild/build-downloads.py: verify standard features
- mpremote: add support for rfc2217, serial over TCP
- metrics.py: fix esp32 and esp8266 board names after renaming
- change remaining "urequests" references to "requests"

CI:
- ci.sh: build both SAMD21 and SAMD51 boards as part of samd CI
- ci.sh: add functions to check code spelling using codespell
- ci.sh: add mimxrt and samd ports to code size build
- ci.sh: build PICO_W board as part of rp2 CI
- ci.sh: add a H5 board to stm32 CI build
- ci.sh: add ARDUINO_PORTENTA_C33 to RA CI build
- workflows: fetch full history for mpremote workflow
- workflows: add spell check to code formatting workflow
- workflows: add GitHub Action to lint Python code with ruff
- workflows: force use of Ubuntu-20.04 for unix 32-bit builds
- workflows: update esp32 CI to use IDF v5.0
- workflows: bump actions/checkout from 3 to 4

The ports
=========

all ports:
- use extmod version of mktime instead of port-specific one
- remove os.sync() implementation from stm32 and renesas-ra
- enable os.sync() for esp32, esp8266, rp2, mimxrt, samd51
- standardise docs link in help text
- in machine_i2s.c, rename uasyncio to asyncio
- simplify board feature tags in board.json
- */boards/*/board.json: remove "id" field
- restrict board.json to standard features
- rename Arduino board LED pins to be consistent

bare-arm port: no changes specific to this component/port

cc3200 port:
- mods/modutime: use extmod version of time module
- tools: fix exception raised on process failure
- Makefile: build firmware.zip

embed port: no changes specific to this component/port

esp8266 port:
- modutime: use extmod version of time module
- add support for the Espressif ESP-NOW protocol
- machine_pwm: implement duty_u16() and duty_ns() for consistency
- add support to set/get power saving mode of WLAN
- change network.WLAN from a function to a type
- allow Ctrl-C to interrupt the corrupt-fs while loop
- machine_pin: accept an integer argument to mp_obj_get_pin_obj
- add board variant support
- boards/ESP8266_GENERIC: add image filename
- boards: make sure modespnow.o is placed in irom0
- boards/ESP8266_GENERIC: remove urllib from the 2MiB manifest

esp32 port:
- modutime: use extmod version of time module
- add support for the Espressif ESP-NOW protocol
- add support to set/get power saving mode of WLAN
- change network.WLAN from a function to a type
- boards: add some missing board configs for two UM boards
- esp32_ulp: enable FSM ULP for S2 and S3 chips
- uart: use xtal as UART clock source on S3 and C3
- modespnow: change name of buffer size config option to "rxbuf"
- CMake: change PROJECT_DIR to CMAKE_CURRENT_LIST_DIR
- esp32_ulp: fix ULP (FSM) support for S2 and S3
- allow Ctrl-C to interrupt the corrupt-fs while loop
- switch from UART driver to UART HAL
- ppp_set_auth: add pppapi_set_auth from ESP-IDF
- modesp32: remove esp32.hall_sensor function
- update port to support IDF v5.0.2
- in recv_cb, get espnow rssi from recv_info->rx_ctrl
- network_wlan: wait for WIFI_EVENT_STA_START after activating
- Makefile: provide more IDF shortcuts
- boards: change SDK config parameters from deprecated to new ones
- modules/inisetup.py: format partition as FAT if its label is ffat
- machine_uart: always select a source_clk value in UART config
- re-enable mDNS after move to IDF v5.0.2
- boards/GENERIC_OTA: enable silent checks to reduce firmware size
- network_wlan: wait for STA/AP START/STOP event in wlan.active
- machine_timer: switch from legacy driver to timer HAL
- machine_pin: add a pin-find func and use it in machine_pin_get_id
- use always machine_pin_get_id for getting a Pin id
- add support for board-named pins and the Pin.board dict
- collect properties from IDF-managed components as well
- modmachine: add generic machine.bootloader()
- usb: add custom TinyUSB callback support
- boards/ARDUINO_NANO_ESP32: add support for Arduino Nano ESP32
- CMakeLists: enable multiple extra component directories in build
- boards/ARDUINO_NANO_ESP32: fix deploy instructions
- main: remove unused mbedtls debug function
- machine_wdt: allow feeding WDT from threads
- machine_hw_spi: fix access of SPI(2)
- machine_hw_spi: remove unnecessary duplicate SPI pin defaults
- machine_hw_spi: remove SPI host renaming for C3 and S3 variants
- machine_hw_spi: check for valid SPI id in constructor, not init
- boards: remove references to the IDF version in board.md files
- README: specify that only IDF v5.0.2 is supported
- allow malloc() to allocate from SPIRAM
- enable automatic Python heap growth
- gccollect: make level arg volatile to force recursive function
- Makefile: implement `make submodules` to match other ports
- boards/GENERIC: merge with GENERIC_{SPIRAM,OTA,D2WD,UNICORE}
- boards/GENERIC_C3: merge with GENERIC_C3_USB
- boards/GENERIC_S2: merge with ESP32_S2_WROVER
- boards/GENERIC_S3: merge with GENERIC_S3_{SPIRAM,SPIRAM_OCT}
- partitions.csv: rename to partitions-4MiB.csv
- use uppercase variant names
- Makefile: append board variant to BUILD
- rename GENERIC* boards to ESP32_GENERIC*
- boards/ESP32_GENERIC_C3: enable UART REPL
- modsocket: add support for SO_BROADCAST socket option
- modnetwork: add support for SO_BINDTODEVICE socket option
- support JTAG console, free up UART
- machine_uart: release GIL for blocking reads
- boards: add pins.csv to UM boards and other minor changes
- boards/UM_NANOS3: add new UM NanoS3 board
- network_ppp: block after deleting task
- boards/ARDUINO_NANO_ESP32: clarify recovery instructions
- boards/UM_FEATHERS3: fix I2C pins in pins.csv
- skip validation of image on boot from deepsleep
- machine_pin: fix null pointer access in machine_pin_find
- mphalport: add function/line/file info to check_esp_err exception
- fix Partition.writeblocks() partial write corruption
- boards: fix VBAT voltage calculation for UM S3 boards
- boards: add bootloader rollback support for all builds
- main: allow a board to override the MicroPython task stack size
- boards/ARDUINO_NANO_ESP32: use Arduino USB IDs
- boards/manifest.py: freeze aioespnow into firmware by default

mimxrt port:
- modutime: use extmod version of time module
- machine_pwm: start PWM only if freq and duty are set
- flash: separate low level driver code from flash object
- mpconfigport: add back lost uos.urandom()
- add missing UART defintion and remove obsolete config
- machine_spi: ignore transfers with len=0
- machine_pin: perform full configuration in machine_pin_set_mode
- sdcard: fix GCC 13 build error with sdcard_cmd_set_bus_width
- led: add support for up to four LEDs
- boards/ADAFRUIT_METRO_M7: add Adafruit Metro M7 board definition
- machine_pwm: fix freq change, PWM print, and error checks
- Makefile: use a specific fsl_flexspi_nor_boot.c for mimxrt1062
- machine_pin: fix bug when Pin.irq is called without a handler
- hal/pwm_backport: fix 0 and 65536 edge cases of PWM's duty_u16
- machine_uart: add uart.deinit method and machine_uart_deinit_all
- machine_uart: add support for UART hardware flow control
- boards: add support for GPIO control of SNVS pins
- hal: make flash clock frequency configurable
- fix UART RTS/CTS assignments for the OLIMEX and Adafruit boards
- machine_pin: extend pin configuration functions
- sdio: add SDIO driver
- integrate support for WiFi via the CYW43 driver
- integrate Bluetooth support with NimBLE bindings
- irq: move all IRQ related definitions to dedicated header
- machine_uart: fix and complete UART.deinit and uart_deinit_all
- boards: fix use of MICROPY_HW_SDRAM_AVAIL in MIMXRT1176.ld
- machine_uart: support slow baud rates for UART
- machine_uart: add a helper function to change the baudrate
- sdio: move config guard so headers are only included if used
- Makefile: update to work with latest TinyUSB
- mpconfigport: don't override parse chunk alloc
- sdio: add support for the 117x series
- mimxrt_sdram: allow boards to override the default SDRAM config
- Makefile: enable the FSL USDHC for supported MCU series
- remove SDCARD Makefile config option
- mpbthciport: enable flow control for BT HCI UART
- mbedtls: enable certificate validity time validation
- machine_uart: set the UART clock to a fixed 40MHz value
- boards/MIMXRT1176_clock_config: fix comments about UART clocks
- boards: fix naming of SD-card config option
- mpbthciport: allow disabling UART flow control for BLE
- machine_rtc: improve the RTC init at boot

minimal port: no changes specific to this component/port

nrf port:
- modules/utime: use extmod version of time module
- boards: rename all nRF boards to use uppercase

pic16bit port: no changes specific to this component/port

powerpc port:
- mpconfigport: don't override parse chunk alloc

qemu-arm port: no changes specific to this component/port

renesas-ra port:
- change MICROPY_HW_BOARD_NAME definition to product name
- modutime: use extmod version of time module
- update boards and ra directory files to support FSP v4.4.0
- add a macro definition to avoid compile error of FSP v4.4.0
- irq: fix typo in comment about IRQ priorities
- consolidate hal_entry.c code and remove hal_entry() func
- boards/make-pins.py: fix PA/PB pins support
- consolidate all fsp_cfg header files to one location
- support changing baudrate for UART
- add support for RA6M5, and add machine PWM, DAC, SDCard
- boards/VK_RA6M5: add new board definition
- remove duplicate machine module from constants list
- machine_spi: consistently use machine_pin_find to get pin
- boards: remove unreachable code in make-pins.py
- Makefile: generate binary firmware output
- add TinyUSB support
- add Bluetooth support using NimBLE
- add RNG driver
- add networking support using lwIP
- add mbedTLS support
- fsp_cfg: add common FSP config files
- boards/ARDUINO_PORTENTA_C33: add support for Portenta C33
- boards/ARDUINO_PORTENTA_C33: update WiFi config
- tune lwip buffers and timing to improve network performance

rp2 port:
- modutime: use extmod version of time module
- machine_pwm: enable keyword args in constructor and add init method
- machine_pwm: add support for inverting a PWM channel output
- machine_pwm: add duty_x() checks and return 0 if PWM is not started
- make rp2_state_machine_exec accept integers
- CMakeLists: allow relative MICROPY_BOARD_DIR when invoking cmake
- mphalport: only use CYW43 MAC for WLAN0 interface
- CMake: normalize MICROPY_PORT_DIR
- add Bluetooth support via cyw43
- mpbthciport: cancel existing alarms
- boards/PICO_W: enable Bluetooth Low Energy support
- machine_pin: factor out pin-find code from machine_pin_make_new
- use uppercase variant names
- Makefile: append board variant to BUILD
- rename PICO, PICO_W to RPI_PICO, RPI_PICO_W
- machine_timer: fix printing of timer period
- mpbthciport: switch to static scheduler nodes
- mpbthciport: fix HCI UART config
- mpconfigport: disable BLE locking when MICROPY_PY_BLUETOOTH enabled
- boards/ARDUINO_NANO_RP2040_CONNECT: use standard HCI UART baudrate
- modmachine: fix lightsleep while wifi is powered off
- msc_disk: allow configuring the USB MSC inquiry response
- README: fix name of RPI_PICO_W board
- CMakeLists: enable debug symbols in all builds
- implement time.time_ns with time_us_64 so it has us resolution

samd port:
- modutime: use extmod version of time module
- machine_pwm: add init() method to PWM and simplify the PWM code
- mpconfigport: drop support for SoftSPI max speed
- boards/MINISAM_M4: update pins.csv for the Mini SAM M4 board
- rearrange the MCU-specific loader files
- mcu/samd51: enable MICROPY_ENABLE_EMERGENCY_EXCEPTION_BUF
- boards: add default deploy instructions
- main: fix sercom deinit ordering in soft-reset
- modmachine: add machine.deepsleep as alias of machine.lightsleep
- modmachine: make some machine classes configurable by #defines
- boards/SEEED_WIO_TERMINAL: rename two pins starting with a digit
- mcu: reduce the startup time after hard reset
- machine_uart: add support for UART hardware flow control
- boards/ADAFRUIT_METRO_M4_EXPRESS: add Metro M4 Express Airlift
- boards: rename flash pins consistently for QSPI and SPI
- samd_spiflash: add SPI flash driver and configure it accordingly
- samd_qspiflash: add QSPI flash driver and configure it accordingly
- adapt existing samd.Flash and integrate with (Q)SPI flash in boot
- boards: extend the code size limit for boards with external flash
- set the LFS type in mpconfigmcu.mk instead of mpconfigboard.mk
- Makefile: move the math lib files from mpconfigmcu.mk to Makefile
- Makefile: print memory region usage instead of totals
- mpconfigport: enable DHT and onewire drivers on all MCUs/boards
- mpconfigport: provide the platform module
- modules/_boot.py: add /lib to sys.path
- mpconfigport: don't override parse chunk alloc

stm32 port:
- modutime: use extmod version of time module
- boards/ARDUINO_PORTENTA_H7: enable FDCAN
- boards/stm32h723_af.csv: fix ADC AF definitions
- mboot: fix alignment of packed final buffer
- irq: fix typo in comment about priorities
- usb: fix USB support on STM32G4
- boards/NUCLEO_G474RE: make it easier to enable USB
- adc: add support for STM32L4A6 MCUs
- boards/NUCLEO_L4A6ZG: add new board definition
- help: don't include unavailable features in help()
- help: exclude legacy entries from help, and adjust constant names
- mpconfigboard_common: provide default spidev config
- boards: use default spibdev config where appropriate
- Makefile: pass relevant CPU flags to assembler
- make-stmconst.py: support structs with names ending in _t
- main: start UART REPL as early as possible
- usbd_conf: treat G0 USB periph as MICROPY_HW_USB_IS_MULTI_OTG=0
- machine_adc: handle ADC resolution less than 8 bits on all MCUs
- add initial support for H5 MCUs
- boards: add ld, af.csv and hal_conf_base.h files for H5 MCUs
- octospi: add preliminary support for OCTOSPI peripheral
- boards/STM32H573I_DK: add H5 board definition files
- add USB support for STM32L1 MCUs
- modify RCC->APB2ENR directly instead of HAL API
- modmachine: remove duplicate machine_timer_type declaration
- modmachine: make machine_reset_cause_obj public
- dac: fix dac.write_timed on G4 MCUs to use 32-bit DMA access
- adc: fix ADC clock prescaler for G4 MCUs
- adc: fix pyb.ADCAll.read_core_temp for G4 MCUs
- adc: fix reading internal ADC channels on G4 MCUs
- machine_adc: fix machine.ADC to work on G4 MCUs
- adc: add workaround for ADC errata with G4 MCUs
- adc: fix pyb.ADCAll.read_core_bat on G4 and L4 MCUs
- qspi: allow qspi_write_cmd_data to write cmd with 1 data byte
- mpconfigport: always define MICROPY_SOFT_TIMER_TICKS_MS
- remove duplicate machine module from constants list
- boards/B_L072Z_LRWAN1: add pin definitions for internal SX1262
- boards/B_L072Z_LRWAN1: lower default ROM level to "Core"
- spi: add STM32WL5 SUBGHZ SPI peripheral
- powerctrlboot: support STM32WL system clock from HSE+PLL
- dma: fix DMA completion on WL55 boards
- subghz: add STM32WL55 subghz radio interface to stm module
- modstm: add MICROPY_PY_STM_CONST flag, clear it for STM32WL5
- use uppercase variant names
- Makefile: append board variant to BUILD
- mboot: fix fwupdate by replacing zlib with new deflate module
- uart: fix UART timeout issue with low baudrate on G4 MCUs
- timer: fix deadtime config on Advanced Timer peripherals
- dma: add support for SPI1 DMA on H5 MCU's
- octospi: add support for dual-line SPI interface
- powerctrlboot: allow using PLL3 for USB clock on H5 MCU's
- timer: fix use of TIM8 on H5 MCU's
- uart: add support for UART10 when it's a USART
- powerctrlboot: allow PLL1 Q and R outputs to be enabled on H5
- adc: fix STM32H5 support
- adc: add support for STM32H5 ADC2 inputs
- adc: optimize sampling time for G4, H5, L4 and WB MCUs
- machine_adc: fix and improve STM32H5 support
- dac: add STM32H5 DAC support, with dma_nohal implementation
- boards: move includes to after defines in all hal_conf.h files
- uart: generalise UART source clock calculation for H5 and H7 MCUs
- dma: remove unbalanced )
- usbd_msc_interface: allow configuring the MSC inquiry response
- boards/ARDUINO_GIGA: update board config
- i2c: add support for I2C4 on H7 MCUs

teensy port: no changes specific to this component/port

unix port:
- modutime: use extmod version of time module
- Makefile: allow variants to add QSTR_DEFS
- main: add NLR scope for checking module __path__
- README: fix Markdown link markup
- modsocket: add poll support for missing ERR,HUP,NVAL poll values
- modselect: remove unix-specific implementation of select module
- modjni: fix build errors with type definitions and error strings
- main: fix memory leakage if MICROPY_USE_READLINE is disabled

webassembly port:
- modutime: use extmod version of time module
- make mp_js_do_str asynchronous
- make mp_js_process_char asynchronous
- replace typeof window check with ENVIRONMENT_IS_NODE flag

windows port:
- Makefile: allow variants to add QSTR_DEFS

zephyr port:
- modutime: use extmod version of time module
RetiredWizard pushed a commit to RetiredWizard/micropython that referenced this issue Mar 14, 2024
Add Adafruit Feather ESP32-C6 4MB Flash No PSRAM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Feature requests, new feature implementations
Projects
None yet
Development

No branches or pull requests