Way beyond my own competence but I've heard bad things about submodules on GitHub. Do you guys have experience of using them?

Submodules are kind of nasty, but subtrees are better: https://developer.atlassian.com/blog/2015/05/the-power-of-git-subtree/

We already use submodules in this repo, for lib/axtls, lib/libffi and lib/lwip.

I think GitHub is fine with submodules, it just doesn't generate clickable links for submodules which aren't themselves on github?

Regarding multiple vs single, the answers here sum it up pretty well: http://programmers.stackexchange.com/questions/161293/choosing-between-single-or-multiple-projects-in-a-git-repository. I've tried both, usually without submodules but using tools like http://linux.die.net/man/1/mr instead. In the current project I'm working on (10+ seperate repositories) we found the benefits (mainly having a shorter, clearer history dedicated to the repository's core business and being able to work on a single one at a time without having to bother what the rest is doing, less merge conflicts) outweigh the cost of managment and tracking down when problems occur. Management problems are basically down to zero by using some small automation scripts, while problems with tracking down bugs and figuring out which commits to seperate repos logically belong together are reduced by having CI and having commit messages reference each other when needed or using tags (so when it does happen that a large breaking change is introduced, all commits would say 'breaking change [SHA-1 of core repo]: Fix' or they all get tagged).

tldr; I'm probably biassed but I'm +1 on multiple repositories and having the core repo as submodule of the port repo seems the cleanest.

@stinos thanks for the input. I think having new ports in their own, separate repo will help us ensure that ports are decoupled as much as possible from the core, and can exist independently. It would also allow to have other contributors with push rights to the ports, the ones that they are interested in maintaining.

yes access management is another advantage, didn't even think of that.

+1 on having separate repos for new ports.

On Wed, Mar 16, 2016 at 4:11 PM, stinos [email protected] wrote:

yes access management is another advantage, didn't even think of that.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#1907 (comment)

• +1 for splitting the repo into smaller ones.
• -10 for including micropython repo into port repos as a submodule. Micropython itself has much faster development than the port code (which is usually just some kind of HAL form the MCU vendor) and it would become very tedious to update submodule everytime Micropython repo changes.

There is an option is to use repo to work with multiple git repositories and combining them into a project.

Maybe @honzik666 can share some insight? He told me about his idea how this could work.

Micropython itself has much faster development than the port code (which is usually just some kind of HAL form the MCU vendor) and it would become very tedious to update submodule everytime Micropython repo changes.

Actually, because uPy core updates often I see it as a benefit for a port to reference uPy via a git-hash using a submodule. Say we have 20 ports which all use latest uPy core. Then it starts to become very difficult for us to make even small changes, because all changes must be tested against 20 ports so that we don't break anything. If we use submodules then ports can move at their own pace and they can update to a new uPy core when it suits them (and when any breaking changes have been resolved).

@dpgeorge I see. What you say makes sense to me now. Thanks for your explanation.

@dpgeorge, @prusnak this is exactly what we have been researching lately - a side product qstr-autogen has come out of it. The main burden of the whole micropython is the buildsystem that is hard to maintain and cover other external components with it. We now have a working prototype where uPy core has been separated from a HAL and the buildsystem has been replaced. We have written a HAL for FreeRTOS (this is not relevant, it can easily be a micropython-stmhal). The main benefit of this layout is a new build system that allows configuring all components in a very Linux - like manner. Unfortunately, it still uses make for the time being with plans to replace it with scons (python building micropython ;-)

@dpgeorge - to address the issue of multiple ports using a certain flavor of the core - this is not a problem at all, I just don't see if git submodules are the best choice. We have very positive experience developing an embedded project for one of our customers and providing regular releases based on a 'repo' manifest. Repo is the standard tool used by Android development and I believe it is much more powerful than git submodules.

+1 for submodules. For the reasons already mentioned. Justifies a "radical" change IMO.

Do you consider to put the /drivers into a separate submodule as well?

Do you consider to put the /drivers into a separate submodule as well?

Not at this stage. Drivers will mostly be in pure Python and therefore small and not a burden to keep in this repo. But of course we can discuss this point :)

But of course we can discuss this point :)

The reason I was asking: drivers (specifically sensors) are also a "HAL" like the HAL of an MCU. If the intention is to get a "clean" bare uPy repo, then /drivers somehow does not fit there - even if it contains not a lot of code.

Coming back to this discussion after about 1 year... what has remained the same is that there is still a lot of work on the core (py/, extmod/, lib/) and I don't think this will reduce any time soon. And it has been very useful to have all the main ports (stmhal, esp8266, unix, etc) together in the same repo, so that changes can be applied across all ports at once, and testing (building all ports to make sure they work with changes, or checking code size changes) is much easier.

Taking experience from the esp32 port (which is currently a separate repo) I think it would be better to keep everything together in the one repo (eg a lot of the esp32 commits are "merge upstream", and contributions/issues in that repo sometimes belong here, upstream).

The main concerns raised at the beginning of this issue are:

1. having a sustainable way to add new ports
2. keeping this repo lean/minimal

To address point 1 I think #2401 is the right approach, to move all ports to a "ports/" directory. I already tried it and it works ok, and 7d4a2f7 was committed as part of that test (that commit should have been done anyway). In the near future I'd like to see esp32, nrf and bare-rpi ports make their way into this repo, and having a ports/ directory will allow to manage that and not clutter the root directory of the repo.

To address point 2 I'd like to move the stmhal HAL libraries to a separate repo and include that as a submodule in this repo. That will allow to easily update the HAL without making a lot of (git) noise in this repo. Furthermore, the HAL is not needed for any other port except stmhal and so users don't need to clone it if they don't use it.

Looking in the current code size of the repo we have (in kbytes):

28      ./bare-arm
40      ./qemu-arm
48      ./minimal
52      ./mpy-cross
60      ./pic16bit
144     ./windows
152     ./zephyr
196     ./examples
220     ./tools
332     ./unix
448     ./esp8266
564     ./extmod
700     ./teensy
712     ./lib/cmsis
1584    ./lib
1596    ./docs
1612    ./drivers
1744    ./logo
2144    ./py
2744    ./cc3200/hal
4484    ./tests
4696    ./cc3200
9132    ./stmhal/hal
14728   ./stmhal/cmsis
26984   ./stmhal
34472   ./.git
82388   .

Out of 82M, 34M is the git history and 48M is the actual source code. Of that 48M, 24M (or 50%) is the stmhal HAL and CMSIS files.

So I think it's a good step to make the ST Cube HAL and ST CMSIS files (but not the files in lib/cmsis) a git submodule.

Most ports will then be under 1M in size, and new ports will hopefully be similarly minimal.

Note that there is already precedence in this repo for having the SDK for a given port as a separate component: esp8266 and zephyr have their SDK installed separately (same goes for the esp32).

In summary, I'd like to:

1. move stmhal/hal/ and stmhal/cmsis/ to a single, separate repo that becomes a submodule under lib/stmsupport (so that all submodules live in the lib/ directory)
2. then move all ports to ports/

I believe, this makes total sense. In fact, a lot of people only use the upy core anyway. I have started a scons initiative here: https://forum.micropython.org/viewtopic.php?f=15&t=3464 and had a talk about this at pycon2017 in Prague.

@dpgeorge I know you have previously stated that there is no big reason to change the build system from make to anything else. I am ok with that, just want to share my experience with scons through this 'pila' scons tool.

+1

As a sidenote to consider: renaming stmhalto stm32?

As a sidenote to consider: renaming stmhalto stm32?

Perhaps... it was originally called "stm" but changed to "stmhal" when the HAL was incorporated.

I already tried it and it works ok, and 7d4a2f7 was committed as part of that test (that commit should have been done anyway).

I definitely agree with that, and it was on my todo for a long time too, since 1e77e25 to be exact. In that commit I however either missed $(TOP), or wasn't sure if it's "top enough" (the idea should be that it should allow out-of-tree ports, modules, etc.), so I introduced $(MPTOP).

To address point 1 I think #2401 is the right approach, to move all ports to a "ports/" directory.

That was also pretty much agreed on, just waiting right time (2.0 release IIRC).

Taking experience from the esp32 port (which is currently a separate repo) I think it would be better to keep everything together in the one repo (eg a lot of the esp32 commits are "merge upstream", and contributions/issues in that repo sometimes belong here, upstream).

That's well expected experience, isn't it? Of course, it "would be better" to keep everything together, as "whole world in a jar" projects like Google Android show. Well, it's better, until various other scalability issues pop up. There's also a solution for "merge upstream mess" - it's rebased-based workflow, which of course has its own issues too. Again, it's all well-known and expected, and it's also well known which workflow to use to optimize for particular outcome (e.g. long-term maintenance vs spur-of-the-moment, extensive growth).

Looking at esp32 fork, it has many issues starting from the most basic ones:

1. Codestyle issues.
2. Missing to route changes to upstream (even though @dpgeorge tries to, some weren't routed IIRC).
3. Wild, extensive feature stuffing without proper requirements/design phases (all very reminding esp8266 or other ports of course, it's just esp32 port is fresh and "right at time", so it's order of magnitude more than in esp8266).

Those all are again expected, and by now should be known by maintainers (which would be expected to do something about), but still happens. Of course, it's not underestimated that new ports also happen, so it's the matter of a balance as usual.

In the near future I'd like to see esp32, nrf and bare-rpi

Why suddenly these at all? We have #1249 as (IIRC) the oldest unmerged port, then #2501, #2634.

What about cc3200 port, shouldn't it be removed by now?

esp32, nrf and bare-rpi

All these ports have issues:

1. esp32 - mention above
2. nrf - (speaking of Nordic port for nrf51822, nrf52832 and nrf52840 #3137) is almost-1000 commit code drop. For starters, number of commit would need to be decimated, then perhaps reduced to 20-30 further.
3. bare-rpi - GPL-based, IMHO with this, we should be strict and not let in (enough of cc3200 with its pseudo-GPL FreeRTOS). It's also a very young one-man project.

As for many other things, there should be formal criteria for getting a port in (compliance with mainlines rules, verified maintenance, etc.). And that's on top of "whether maintainers want to maintain it" - without it, it's an immediate bitrot (again, we have cc3200 as a guinea pig for that).

move stmhal/hal/ and stmhal/cmsis/ to a single, separate repo that becomes a submodule under
lib/stmsupport (so that all submodules live in the lib/ directory)

then move all ports to ports/

Sounds good, the question is when. If you say "right away", it's probably ok (well, re: #3270), but point 2 of that shouldn't be taken as opening the gate for new ports waterfall, as that IMHO would take much more preparation (mostly on the part of candidate ports, again IMHO).

Why suddenly these at all? We have #1249 as (IIRC) the oldest unmerged port, then #2501, #2634.

Because esp32 is now becoming widely used, nRF port is long overdue and a bare-metal port to RPi is something new and interesting and shows further how uPy can be used.

What about cc3200 port, shouldn't it be removed by now?

It still works, and as far as I know people still use it. So why not keep it?

As for many other things, there should be formal criteria for getting a port in

Too many criteria and rules stifles innovation and slows progress. As usual we disagree on a lot of the management points.

Sounds good, the question is when.

After tagging v1.9.2 it would be good to move forward with these plans here. But I don't think it's worth making it a v2.0 change: v2.0 can come after the dust settles.

Too many criteria and rules stifles innovation and slows progress.

And too little leads to a mess, which slow progress to a halt, then there's hardly any room for innovation at all.

As usual we disagree on a lot of the management points.

Yep, or alternatively, there're not enough parties who care about organization and cleanness, so I'm filling that role ;-).

The ST HAL libraries have been moved to a submodule, and all ports have been moved to the ports/ directory. So this issue can be closed.