PEP 738: Adding Android as a supported platform

hugovk · 2023-12-13T20:51:06.291Z

This bit of PEP 12?

Add the direct URL of the PEP’s canonical discussion thread (on e.g. Python-Dev, Discourse, etc) under the Discussions-To header. If the thread will be created after the PEP is submitted as an official draft, it is okay to just list the venue name initially, but remember to update the PEP with the URL as soon as the PEP is successfully merged to the PEPs repository and you create the corresponding discussion thread. See PEP 1 for more details.

Yes, that can be clarified, and good point about the pre-commit check.

Julian-O · 2023-12-14T00:50:47.611Z

[Newbie here. Hope I am posting in the right place.]

In the PEP 738 draft, there is a section on platform, and in particular the platform.android_ver() namedtuple.

It defines the model field as

the model name of the device, as a string (e.g. "Pixel 7").

I think this definition may need more thought.

If we look, for example, at this list of Android devices we see “Pixel 7” appears as a Marketing Name and a Model.

As I understand it, the marketing name is not available through the Android SDK. (My reference for this is pretty weak, so I would be happy to be contradicted.)

At the same time, the model may be ambiguous, with lots of devices having generic names like “4K SMART TV”.

I think the definition of the model will need to use some sort of concatenation, with smarts to avoid duplicating the manufacturer’s name if it appears in the model, similar to the suggestions on StackOverflow question linked above (I have reached the newbie link limit, sorry.)

As an aside: I note that models may be useful for techie humans to identify devices, they aren’t very useful for making programmatic decision, because they are unpredictable.

e.g. they are good for “Oh, there’s the problem! I am connecting to my old Android phone, not my new Android TV”, but they are not good for “The Samsung Galaxy S6 series has a problem with its camera, so adjust the exposure if the model is SC-05G, SM-G920F, SM-G920I, SAMSUNG-SM-G920AZ, 404SC, or any of about 50 other model names.”

freakboy3742 · 2023-12-14T04:21:34.602Z

Full disclosure: I’m @mhsmith’s co-worker, so it won’t be surprising when I say I’m fully in favor of this proposal .

For some background: BeeWare has supported Android for ~9 years; but that support has been in 4 distinct phases:

An initial “embedded CPython” approach, many years ago. That was abandoned because of kernel-based JNI limits that existed on Android in the early days (Pre Android 4)
An approach based on transpiring CPython code to Java byte code. This didn’t embed any C code, but was effectively a from-scratch Python re-implementation. It allowed you to write an Android app in Python, but it didn’t allow interaction with most of the Python ecosystem (essentially anything with a C module, or anything in the standard library that was implemented in C)
An “embedded CPython” approach that BeeWare maintained. This is part of the the work that the PSF funded in 2019.
Using Chaquopy’s embedded CPython build and related tooling.

What has been proposed in this PEP is consistent with my experience from those implementations. For the most part, this is really a “can we just run Android in CI and formalise this already?” PEP. Unlike iOS, the number of extra patches needed is relatively small - it’s mostly changes around documentation and CI integration.

However, there are a couple of novel parts in the PEP for which I want to formally voice my support.

Specifically, I’m a strong +1 on the sys.platform = 'android' part of the proposal. Having sys.platform='linux' on Android is a “technically correct is the best kind of correct” answer. Yes, Android is, technically, “a Linux”. In practice, while there is some “Linux compatible” code that will run unmodified on Android, there’s just as much that isn’t. This is because the things that sys.platform is used to gate are, by definition, platform specific. In practice, “Linux” isn’t just a kernel - it’s lots of other device expectations, filesystem expectations, userspace expectations, and more.

At the very least, the build process for C modules is almost always radically different, and there’s plenty of setup.py build configurations that use sys.platform as a way to identify platform-specific compilation flags - which are rarely the same on Android as they would be on a Linux.

As a simple matter of practicality, being able to do dictionary lookups on sys.platform to select Android-specific logic, and use sys_platform=='android' in a requirements specifier are two basic UX affordances that are worth their weight in gold. The price we pay for this affordance is that code that would run unmodified will require a patch to read if sys.platform in {'linux', 'android'}. For me, this is a throughly acceptable and reasonable price to ask.

I’m also in favor of the changes to platform that expose more “as experienced by the user” values to uname(), etc. However, I suspect these will be less controversial.

freakboy3742 · 2023-12-14T04:22:47.296Z

Also, for visibility: I’ve circulated this PEP with the Kivy team; hopefully one of them will weigh in with their thoughts.

Julian-O · 2023-12-14T12:58:54.374Z

[Caution: I am largely ignorant about what I am about to say, and would be happily fobbed off with “Yeah, don’t worry yourself. That’s not a problem.”]

There is a line in the Buildozer documentation:

Also, because the Android platform cannot load more than 64 inline dynamic libraries, we have a mechanism to bundle all of them in one to ensure you’ll not hit this limitation.

I do not know if that limit still applies since the Android 6.0 dynamic linker improvements.

I do not know if Android wheels each count as inline dynamic libraries; I assume so.

I do not know if this is a simple packaging issue that will continue to be solved by Buildozer.

I just want to be sure that someone with expertise has considered this, and this doesn’t accidentally become a bottleneck limiting the number of wheels that can be installed.

misl6 · 2023-12-16T14:11:23.165Z

Hi, Kivy Core Dev here!

I’m glad to see that the Python community is already working on the Android counterpart of PEP 730 (that I fanboyed at PyConSE last month , like a child that sees the Xmas present he long-awaited).

Back to this PEP, I re-read it twice and there are just a couple of proposals that I feel can be beveled before these changes can potentially impact our fellow Pythonistas that maintain apps Python apps on Android in specific environments, where looking in the long term is a must.

I’m the kind of dev who tries to remove the additional burden when possible, so I would be happy to simplify things, but if I look at the scenario from a user (and management) perspective things change quite fast.

I’m not involved in any of these scenarios (happily), but I’m aware of Kivy users who still have to maintain compatibility on old devices and Android versions. (And the same happens for “native” Android devs)

So …

For Python 3.13 we propose the minimum Android version to be 6.0 (API level 23). This would support 98% of active devices, and would allow us to rely on a number of dynamic linker improvements which simplify the use of dynamic libraries.

x86 has not been supported as a development platform since 2020, and no new emulator images have been released since then.

armeabi-v7a is still used, but its proportion of active devices is now less than 10% and steadily falling.

That’s (almost) true if we talk about phones and tablets that the majority of the population have in their hands, pockets, or backpacks, but that does not apply to Android devices that we use every day without knowing these devices are powered by Android (and Android apps).

As an example:

EV charger stations
Payment terminals
VOIP phones
Self-service machines
Smart Home screens
… and many more

All these devices have a longer lifespan, compared to smartphones and tablets (that we usually change every ~4 years), and when they get provisioned for the first time, the hardware and the Android version are usually already obsolete.

Why? (in order of importance):

Availability
Costs
Bad management decisions

What I’m proposing:

For the minimum API version, let’s stick to the one the latest NDK LTS supports. (21 ATM)
For the supported platforms, let’s stick to the ones that the latest NDK LTS supports. (arm64-v8a , armeabi-v7a, x86_64, x86)
At every Python release, check if a new NDK LTS version has been released, and let’s increase the minimum supported API version, and if needed change the supported platforms accordingly).

How we can test 32-bit variants?

64-bit devices also support their 32-bit variants. Using arm64-v8a devices as an example, the device can also run armeabi and armeabi-v7a code.

So, we can produce armeabi-v7a artifacts, and run them on arm64-v8a, and the same applies to x86 artifacts on x86_64.

In years of maintaining the python-for-android toolchain, I did not see anything blocking maintaining armeabi-v7a and x86 artifacts. Usually, when a recipe is correctly written for arm64-v8a, that also works for the other variants.

encukou · 2023-12-18T14:54:46.280Z

Thank you for the concern. And for Kivy, too!

I think that it’s OK for this PEP to target relatively modern Android versions. That does mean non-phone devices would be left to third parties for now, but that might actually be good: you know these devices best after all!
And, support for 32-bit and older LTS versions could always be added after the initial Android support is in.

It’d be a very good entry for a “possible future extensions” section in the PEP, though!

@mhsmith, feel free to leave that section to a future PR, so the current PR can be merged soon.

mhsmith · 2023-12-22T22:20:47.715Z

Thanks everybody for your comments. I’ll be on vacation for the next 2 weeks, but I’ll get back on this in the new year, and we can put together a second draft.

mhsmith · 2023-12-22T22:37:57.544Z

Julian:

Also, because the Android platform cannot load more than 64 inline dynamic libraries, we have a mechanism to bundle all of them in one to ensure you’ll not hit this limitation.

I do not know if that limit still applies since the Android 6.0 dynamic linker improvements .

This limit was actually removed even longer ago than that, in Android 4.2 or 4.3. I can confirm that since then, loading hundreds of dynamic libraries into the same process is no problem.

mhsmith · 2024-01-15T18:17:16.187Z

Julian:

As I understand it, the marketing name is not available through the Android SDK. (My reference for this is pretty weak, so I would be happy to be contradicted.)

At the same time, the model may be ambiguous, with lots of devices having generic names like “4K SMART TV”.

That’s a good point. The Android API exposes both a “model” and a “device”. Which one is more likely to be unique, and which one is more likely to resemble the marketing name, varies between different manufacturers.

So I think the best solution is to expose the model, device and manufacturer as three separate fields returned by android_ver.

Julian:

As an aside: I note that models may be useful for techie humans to identify devices, they aren’t very useful for making programmatic decision, because they are unpredictable.

Agreed: the point of providing this information is mainly so apps can include it in bug reports or usage statistics. In which case, it’s better to return the three fields unmodified, rather than trying to do anything clever with concatenation or deduplication.

mhsmith · 2024-02-01T21:32:13.628Z

Mirko Galimberti:

Hi, Kivy Core Dev here!

Hi, it’s great to get you involved. If anyone else in Kivy is working on Android support, please point them at this thread.

Mirko Galimberti:

For the minimum API version, let’s stick to the one the latest NDK LTS supports. (21 ATM)

After further investigation, I agree that API level 21 will be fine. The dynamic linker improvements in level 23 may be necessary for some packages, but they shouldn’t be a blocker for Python itself.

Mirko Galimberti:

64-bit devices also support their 32-bit variants. Using arm64-v8a devices as an example, the device can also run armeabi and armeabi-v7a code.

This is true for physical devices, but one of the requirements of Tier 3 support is a reliable buildbot, and that would be difficult to maintain with a phone plugged into a server somewhere. So the buildbot will have to use an emulator, which we in the BeeWare team have quite a bit of experience with from our own CI (see the “testbed (android)” jobs).

The only ARM platform the Android emulator can run on is macOS, but ARM Macs have no 32-bit support in hardware, so they can only run armeabi-v7a code using software emulation. I tried this on my own MacBook Pro, and the emulator took a very long time to start up, but was reasonably usable after that – until it crashed after a couple of minutes. And that isn’t surprising, because Google stopped updating their armeabi-v7a emulator images in 2016, so I doubt they’re testing them with their current emulator engine anymore.

So to be confident of getting everything up to the Tier 3 standard during the Python 3.13 cycle, I think we’ll need to limit our focus to the 64-bit ABIs. But just because the 32-bit ABIs aren’t Tier 3, doesn’t mean that they won’t work. We won’t intentionally break them, and I recommend that PRs should still be accepted for them. In other words, they’ll have exactly the same status that all Android ABIs have had up until now.

mhsmith · 2024-02-08T16:28:37.279Z

The second draft of this PEP is now up for review:

github.com/python/peps

PEP 738: Android support, draft 2

python:main ← mhsmith:android-support-2

opened 04:26PM - 08 Feb 24 UTC

mhsmith

+57 -29

* Change is either: * [x] To a Draft PEP * [ ] To an Accepted or Final… PEP, with Steering Council approval * [ ] To fix an editorial issue (markup, typo, link, header, etc) * [x] PR title prefixed with PEP number (e.g. ``PEP 123: Summary of changes``) Updates following [discussion on Discourse](https://discuss.python.org/t/pep-738-adding-android-as-a-supported-platform/40975): * Reduce minimum Android API level from 23 to 21. * Add more device information fields to `platform.android_ver`. Other changes: * State that a `python3.x` executable will not be officially supported on this platform. For the reasons explained in "App lifecycle", this would be of very little use. * Add another reason why libpython must be dynamically linked. * Add details of stdin, stdout, and stderr. * Remove `spwd` from unsupported modules list, since it's already been removed from Python 3.13. @freakboy3742: FYI ---- 📚 Documentation preview 📚: https://pep-previews--3651.org.readthedocs.build/

mhsmith · 2024-02-13T11:56:26.788Z

This is a last call for comments on the PR above. If there are no remaining issues, I’ll submit the PEP to the steering council later this week.

encukou · 2024-02-13T15:19:14.822Z

I merged the PR; PEP-738 will update in a few moments.

mhsmith · 2024-02-15T11:23:44.596Z

The PEP has now been submitted to the Steering Council.

thomas · 2024-02-29T14:29:44.142Z

(Not posting for the whole SC, although this question did come up during SC deliberations.)

What is, concretely, the plan for buildbots for Android? Tier 3 technically requires a buildbot, not GitHub CI. As we’ve been reminded of just a few days ago, CI doesn’t run all the tests, and it’s easy for unintended breakage to slip in if we only rely on CI. There’s also the matter that ARM64 runners in GitHub’s CI are somewhat of a scarce resource right now, and the tier 2 platforms that use them should have higher priority.

For iOS, PEP 730 mentions Anaconda offered to host buildbots. This PEP mentions using Anaconda resources for CI. Does that mean buildbots? Because I think we really need them – or someone – to host buildbots for both of them.

mhsmith · 2024-02-29T21:11:23.834Z

Yes, I can say on behalf of Anaconda that we will host the buildbots for both iOS and Android. For Android this will likely mean both Linux x86_64 and macOS ARM64 machines, so we cover both runtime architectures and both supported build platforms.

encukou · 2024-03-01T10:28:03.664Z

Would you mind updating the PEP?
You should avoid big changes while the SC is discussing, but this clarification should be fine, with a note in the SC ticket.

mhsmith · 2024-03-01T23:07:56.947Z

OK, I’ve created a PR:

github.com/python/peps

PEP 738: clarify Android CI availability

python:main ← mhsmith:android-ci

opened 11:06PM - 01 Mar 24 UTC

mhsmith

+15 -7

This PR clarifies the following points which were [raised on Discourse](https://…discuss.python.org/t/pep-738-adding-android-as-a-supported-platform/40975/19): * The availability of buildbots and GitHub Actions resources for testing Python on Android. * That the supported build platforms for Android will be Linux and macOS. * Change is either: * [x] To a Draft PEP * [ ] To an Accepted or Final PEP, with Steering Council approval * [ ] To fix an editorial issue (markup, typo, link, header, etc) * [x] PR title prefixed with PEP number (e.g. ``PEP 123: Summary of changes``) ---- 📚 Documentation preview 📚: https://pep-previews--3700.org.readthedocs.build/

thomas · 2024-03-20T15:52:22.707Z

The SC has accepted the PEP: PEP 738 -- Adding Android as a supported platform · Issue #231 · python/steering-council · GitHub