Lua and Python

By Jake Edge
February 11, 2020

From a high-level perspective, Lua and Python are similar languages; both are "scripting" languages that are compiled into bytecode instructions that run on a virtual machine. But the focus of Lua has generally been toward embedding the language into some larger application or system, rather than as an alternative for, say, Python, Perl, or Ruby as a general-purpose language. That is not to say that Lua is not capable of handling any of the tasks those other languages do, but that it has not really been the target, seemingly. Some recent discussions in the Lua community have explored possible changes in that regard, particularly around the idea of providing a larger, richer standard library.

In mid-December, Gavin Holt posted a message to the lua-l mailing list noting his overall happiness with the language, but observing that support for the "batteries" was lagging. The "batteries" term presumably comes from Python, which has long had a "batteries included" philosophy—it ships with a large standard library that is meant to cover many of the tasks a user might encounter, without needing to install additional packages. As Lua has progressed through 5.1, 5.2, and 5.3 (with 5.4 on the horizon), however, some of the packages have not been updated, so Holt cannot use them with recent releases of the language. Many of the Lua libraries are written in C, which need to be compiled and distributed as binaries for some users.

Lua is well suited to the "non-compiling user", as are many other interpreted languages. The popularity of these scripted languages is in part the ability to use them without learning/maintaining a compiler. However, there is a ceiling to what can be achieved with Lua alone and from that point onwards you need compiled libraries aka "Batteries" (lfs, winapi, rex_pcre, clipboard, afx, lpeg, hunspell, lsqlite, vcl, gslshell). I am very grateful to those who have made compiled libraries for windows available to download, I don't have or want a compiler!

As might be guessed from the last point, Holt is running on Windows—Linux users may be somewhat more accustomed to having and using a compiler on their systems. "I know the 'non-compiling' user may not be the main target audience for Lua, but as a beautiful interpreted language I feel the lack of 'Live Batteries' prevents Lua becoming as popular as it should." He wondered if there was any effort to put together a distribution of pre-compiled libraries.

There were several suggestions of places to look for the pieces that Holt needs in the thread, but the overall picture is that Lua lacks a standardized set of libraries—which has both good and bad points. Dael Vnaja thought it would be useful to have "an alternative Lua distribution/fork that includes these 'batteries' from the language itself and not from 3rd party libraries", but that the user base for Lua might be too small to sustain one. Coda Highland said that the actual user base was quite large, but that each of the big players has its own requirements: "Everyone has different needs so the places where serious money is getting poured into using Lua don't really have much of an incentive to standardize -- it wouldn't give them any benefit."

Choosing libraries

But there are some problems trying to choose reasonable libraries for Lua, Steve Litt said. First, options need to be found, then they must be installed and tried, which may require multiple iterations. If the library is not packaged by the distribution being used, it needs to be grabbed from elsewhere (possibly using the LuaRocks package manager) and any users of the code need to also get instructions on how to install the library. It is quite different in the Python world:

Contrast this to Python, which has a curated set of complete, tested and documented standard libraries, effectively meaning that almost any project you start in Python can be finished in Python using just the standard library.

[...] Lua is by far the best *language* I've ever used. However, due to lack of a curated and up-to-date standard library, it's not the most likely to produce a successful result in a computer application as opposed to an embedded program.

He suggested that "an official list of approved Lua libraries", which could be searched and that would provide information on the strengths and weaknesses of the library, would be a good place to start. It would be far better than "having to ask online and getting lots of opinions".

Some agreed with that idea, but others were not so sure. Python exists, so perhaps those who need a full-featured standard library should turn to that, Dibyendu Majumdar said. "Lua's strength is lack of any baggage, which makes it better for embedded use." But no one is suggesting that a set of batteries become a requirement for Lua—even the existing Lua standard library can be compiled out for use cases where it is not needed, Lorenzo Donati said. The main reason that Python is a better general-purpose language than Lua is "because of its libraries", he continued.

By all accounts, Lua works quite well in its niche, but some see the potential for it to grow beyond that niche—and the main thing holding it back is the lack of a rich, standard library. As Litt put it:

But given the fact that Lua, as it has existed for the past five years, is a pretty much perfect language, curating a group of approved add-ons would be the best way to improve Lua.

The conversation picked back up again in mid-January when Donati posted some links about the popularity of Python. Stefan Lederer replied that it is not surprising that Python is as popular as it is "because of powerful libraries and easy scripting", but that does not mean the Lua community should be trying to create a "full-fledged Better Version Of Python That Is Actually Lua". He thinks that Python suffers from problems on multicore systems that are only going to become more acute and believes that Lua is better positioned for multicore. That needs more work too, however.

Litt suggested that the existence of its batteries made Python a better choice for a larger project: "If you start a project in Python, you're more likely to finish it in Python as opposed to if you'd started it in Lua and expect to finish it in Lua." He posited two possible directions for Lua, either recognizing that its lack of batteries is holding it back for the general-purpose use case or simply deciding that Python already fills the other role, so Lua should stick with the embedded use case. He was not in favor of the latter:

IMHO in this day and age, you need a language that can easily and reliably and without experimentation interact with XML, HTML(5), Xhtml, databases, JSON, YAML, DOM, sockets, IPC, complex numbers, and probably twenty other things I haven't thought of yet. And with small appliance devices doing ever more mainstream things, some of these batteries will probably become more important, in embedded programming, in the future.

So, in my opinion, Lua would greatly benefit from an official, blessed, curation of libraries available both in a single package for when disk space isn't at a premium, and a-la-carte for embedded.

But a set of standard libraries has its own set of downsides, Peter Hickman said. Languages all have their sets of tradeoffs; Lua is simple and fast, even though it sometimes lacks for a particular library he might need or want.

Although I get pissed when searching for an existing Lua library to do some task to find either zero or six. Not having standard libraries is not necessarily a bad thing. In other languages the standard / default library for something tends to have a kitchen sink set of features just in case someone might need feature X. This comes at a cost, the codebase is larger, it has codepaths that are addressing features you have no interest in, hitting performance and eating memory. More code means more possible bugs. Just look at the standard Python logger. It is truly a marvel of engineering. Of which I use at most 10% but the remaining 90% is baggage my application must carry. Same with Ruby, every unused language feature is hitting performance and eating memory.

Collaboration platform

Oliver Schmidt thought that the missing piece is not really a set of batteries, but is instead a way to collaborate on gathering, rating, and discussing the existing (or new) libraries for Lua. In essence, there is a lack of a Lua community working together to create a code repository that is "useful for newcomers and common use cases". In order for Lua to grow, there is another path should perhaps be considered:

Everyone cooks his own soup that perfectly fits his use case. This is perfectly OK IMHO, this is not a "bad thing", but everyone who wants Lua to have more community impact sees that there is something missing in the current way.

Currently, the LuaRocks site has a bunch of libraries that can be installed, but some important ones are missing and the site lacks many of the features Schmidt thinks are needed. One way forward might be to extend the LuaRocks site, but perhaps setting up a new site on an existing platform (e.g. GitHub) would be an alternative, he said. As became clear in the rest of that thread, though, the community effort required for any kind of solution may not really be available.

Blessing

There is also the question of which libraries would go into some putative richer standard library. Some in the threads seem to think that the Lua team should "bless" a set of libraries, though that has problems of its own. As one of Lua's founders and a member of said team, Roberto Ierusalimschy, put it: "I would break the steps even further: first step is a stable library. Nobody can bless anything unstable."

Another sub-thread took up the idea of a separate repository where the community parks its library code to be improved upon collaboratively; Rodrigo Azevedo proposed the LuaDEAL (for "Lua DEad Alive Libraries") project for creating a kind of standard library. A first step is to identify which libraries fit: "It means, when someone on this list asks about a library, everybody must agree on a first default answer. Public libraries are blessed by the community, not a specific restricted set of people." The choices are not meant to be the "best", just something that is useful to get a particular job done. In addition: "They must have a well defined and stable 'Lua-ish' API. Documentation is mandatory."

There were some discussions about what that API should look like, and even a fair amount of agreement, but the fact remains that the Lua community is small. Dennis Fischer split up Lua users into a handful of categories and noted that Lua enthusiasts would seem to be the smallest of those groups. Growing the numbers of enthusiasts is needed, but there are some problems in the way of doing so:

So the alternative would be to somehow capture more of the users that really only pick up Lua for some very specific task. The problem with this is that, sadly, many Lua APIs are just trash. There's a world of globals, ad-hoc code and poorly translated pythonisms out there, and not many examples of idiomatic, well-designed APIs to offset it all.

That seems to be the crux of the issue: there is no collection of Lua libraries that "everyone" could coalesce around, even if enough everyones could be found. There are tons of Lua libraries available, but finding the "right" one is difficult, such that many users simply go off and implement their own. Even if a library appears to be suitable, it can be hard to find out if will work on the operating system of interest or the Lua version (or alternate implementation such as LuaJIT) needed.

That stands in stark contrast to the Python world, where its large standard library is immediately available to import. Beyond that, add-on libraries are plentiful and generally well-specified on the Python Package Index (PyPI), which can be installed via pip. To a large extent, the idea of what is "Pythonic" is fairly well-understood in the community as well; "Lua-ish" seems to be a bit more amorphous, or at least not well socialized within the community.

Certainly Python has struggled at times with its batteries. Some modules were perhaps mistakenly adopted into the standard library, others have not aged well, and some have even bit-rotted to an extent, which has led the project to discuss removing some "dead" batteries. There are also a fair number of API warts among those modules. But overall, the choice made early on to include a standard library with the language itself has paid big dividends for Python—and its popularity.

Lua is not much younger than Python—Lua came about in 1993, while Python was started in 1989—but Lua's focus has always been different. Trying to change the inertia of the project and its community nearly 30 years down the road is likely to be a difficult task. It is not insurmountable, perhaps, but it will take a great deal of focused effort—one library at a time. Whether the effort is worth the price is hard to judge in the early going as well.

Nothing here is meant to disparage Lua or its community in any way. By all accounts, Lua is an excellent choice for a lot of tasks. Its community seems friendly and welcoming as well—in many ways lua-l is reminiscent of the Python discussion forums. The lack of batteries is not really a new topic in the community, and a variety of solutions have been tried, so it will be interesting to see where things go from here.

Index entries for this article
Python	Other languages

to post comments

Lua and Python

Posted Feb 12, 2020 0:55 UTC (Wed) by dvdeug (subscriber, #10998) [Link] (2 responses)

"Although I get pissed when searching for an existing Lua library to do some task to find either zero or six. Not having standard libraries is not necessarily a bad thing. In other languages the standard / default library for something tends to have a kitchen sink set of features just in case someone might need feature X. This comes at a cost, the codebase is larger, it has codepaths that are addressing features you have no interest in, hitting performance and eating memory. More code means more possible bugs. Just look at the standard Python logger. It is truly a marvel of engineering. Of which I use at most 10% but the remaining 90% is baggage my application must carry. Same with Ruby, every unused language feature is hitting performance and eating memory."

I'm curious about the hard numbers here. It's easy to complain about performance, but if you're writing to a hard drive, ballpark that's going to cost 10,000 times what a memory access costs, so you can add a lot of logging features and have any noticeable speedup or slowdown wiped out when you start writing, or worse, syncing.

I might separate out libraries into slow libraries and speed libraries. Slow libraries are things that involve I/O or are at least run very rarely; one can lose hundreds or thousands of cycles and not notice it, possibly not even be able to measure it. Speed libraries are stuff like math and data structure libraries, and if they're well-developed, they can sometimes run massively better than a naive reimplementation, both in real time and Big-O, as well as being tested. In both cases, I'd take one standard library over six options. (Here is six pure Lua or Lua/C options for linear algebra, or a Lua API to Lapack. I'll take the latter, for speed and tested nature, thanks.)

"More code means more possible bugs" is true in certain contexts, but not, I think, in the case of a standard library versus an ad-hoc toss-off implementation. JWZ has a post about rewriting Mozilla, that makes a point about old code often handling edge cases and bizarre errors that its replacement doesn't, that I think applies to standard libraries versus six alternatives.

In the case of Lua compiled into a powerbank (which apparently has more computing power than the Apollo moon landers), sure, I can see it. (I can also see striping the standard library down for such a use case, or using a minimal standard library.) On obsolete hardware like an iPad 2 or Cray 2 (approximately equal in computing power), I'm not sure. If you're running a Raspberry Pi or bigger, I'm only worried about loading times from slow storage.

Lua and Python

Posted Feb 12, 2020 3:01 UTC (Wed) by smoogen (subscriber, #97) [Link] (1 responses)

I think you are looking at the rare use-case where lua would be used on general usage systems. Most of the current lua use-cases are in embedded systems where gigabytes are expensive in the amount of electricity you have a budget for and you may have a 64 bit cpu but only megabytes of memory. [We aren't talking something luxurious like a flip phone either.. this might be something embedded in a remote sensor or some other constrained device.]

Even when you are on a system with 'loads' of RAM (say a lightbulb or a phone), you are going to be embedded in another program where every extra bit of memory you don't use is something that program can't. This means that anything which looks like bloat is jettisoned.

The context here is important because the original author (and most of the core lua people ) is going to be worried about that scenario. Any 'blessed' library is going to be wanted to be used in those environment (just as much as people want the full python standard ones to be available in micropython or micropython is useless to them).

Lua and Python

Posted Feb 12, 2020 17:43 UTC (Wed) by admalledd (subscriber, #95347) [Link]

This exactly on the common constraints Lua is used under. We use Lua embedded in our own product which does run on 'loads of CPU and RAM', but we use Lua as a safe sandbox (through many sanity/safety layers and <redacted> cheating protection) to run basically user/client provided code when we process their work. One of the _small_ jobs entails executing something on the lines of 100,000 short Lua scripts. We are also constrained on "faster we go more $$" so of course we multithread, but now that means each and every Lua stack, variable, byte of memory used is multiplied.

So we strip even the standard lib (of course things like the `debug` module or `io` are right out for "don't trust the user" reasons!) down to as little as we can allow. A few memory pool tricks here-and-there too.

So even when embedded Lua is used on a big server (woo triple digit cores! 512GB+ ram!) we have to run _multiple_ instances of our process in parallel, and it itself is threaded, so suddenly we do start to care here too.

In the end, I am all for at least better "hey you are trying to read XML/JSON/YAML in Lua? here is are some modules or examples to look at" documentation/lists to exist. It is very rare if ever we developers need to write something new, the challenge is even finding said existing prior work to use/improve even when we want to. Would have made the initial integration less rocky.

Lua and Python

Posted Feb 12, 2020 4:19 UTC (Wed) by re:fi.64 (subscriber, #132628) [Link]

Honestly I do feel like adding some basic stuff like lua-filesystem as part of the standard distribution would not only make the language easier to use, but even make some embedded use cases easier or at least simpler from a build / distribution standpoint.

Lua and Python

Posted Feb 12, 2020 14:22 UTC (Wed) by tau (subscriber, #79651) [Link] (17 responses)

Sounds like they need a package manager like pip, npm, gem, maven, cargo etc.

Each of these projects have rediscovered and converged upon broadly the same design, so we have a good understanding of how to build such a tool at this point.

Lua and Python

Posted Feb 12, 2020 16:35 UTC (Wed) by koh (subscriber, #101482) [Link] (15 responses)

Why is there a need to reinvent package management for every language? The basic task is the same as the distribution's PM. What is so language-specific? My guess is that it is just easier to aggregate all packages using one specific (or one could say "made up") set of packaging rules than to follow a style that makes it easier for distributions to package the software themselves. Of course, granularity is another friction point.

My understanding is that a way forward for the Lua people would be to a) come up with a kind of guide what makes a good "standard library candidate" package. Then, based on that, b) create a comprehensive list of examples that would form said standard library and finally c) try to get these developments packaged/packageable. Instead of writing another package manager. The benefit would be that a) and b) are reusable and referenceable, with a) potentially even containing language-agnostic goals which would make it useful for a broader set of users.

Lua and Python

Posted Feb 12, 2020 17:04 UTC (Wed) by excors (subscriber, #95769) [Link] (10 responses)

There is no distro package manager on Windows or OS X or Android etc, so you need to write your own for your new cross-platform programming language. (You can't copy another language's package manager, because they probably didn't design it to be generic enough, and because they probably wrote it in their own language which you don't understand or like (which is why you made your own language), and because you don't want your language to be weighed down by a dependency on someone else's language.)

And there are too many distro package managers on Linux, so it's a pain to learn the subtleties of all of them and it takes a lot of effort to write your own packages for every one, or else it's a lot of effort to understand and coordinate with the packaging communities of each distro to get them to maintain packages for you. And then distros all have different update schedules and some will be years behind others, so a lot of users will complain they can't use the new language features or libraries that you've written, and will want a way to install updates independently of their distro package manager.

Since you've already had to develop a custom solution for Windows and OS X, it's much easier to use that on Linux too and ignore the distros completely.

Lua and Python

Posted Feb 12, 2020 18:24 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (3 responses)

> There is no distro package manager on Windows or OS X or Android etc, [...]

That's fair.

> And there are too many distro package managers on Linux, [...]

That's a stretch. There are two that you should (maybe) care about: RPM and APT. Everyone else can just fend for themselves as far as I'm concerned. Provide a tarball and let the smaller distros work it out on their own.

> [...] so a lot of users will complain they can't use the new language features or libraries that you've written, and will want a way to install updates independently of their distro package manager.

That applies to all software that anyone has written for any purpose, ever. And I'm only slightly exaggerating.

IMHO the real audience of language-specific package managers is developers. End users are usually going to want a distro-level package (or self-extracting installer, on platforms which lack packages) which may or may not pull in other deps, but should ultimately "just work" without a significant amount of fiddling. They don't necessarily care about getting the latest and greatest, because "just works" is often a much more important property (see for example Debian stable).

Lua and Python

Posted Feb 12, 2020 19:31 UTC (Wed) by tau (subscriber, #79651) [Link] (1 responses)

Language-specific package managers greatly reduce the friction caused by importing third-party libraries. People like to make fun of JavaScript for using libraries containing a few dozen lines of code (that have comprehensive unit tests, non-obvious patches contributed for surprising edge cases, and lots of real-world testing) but this would be utterly impossible in a world where you had to wait for your library to get packaged by Red Hat and Debian, then wait again for Ubuntu to pick it up, and then wait a few years for that package to become available in the stable releases of all three distributions. Low-friction third party library ecosystems are a huge boost for developer productivity.

Also bad third party libraries can fall out of use, bad standard libraries are forever. Standard library code is also subject to the release cycles of its language runtime.

Lua and Python

Posted Feb 13, 2020 8:09 UTC (Thu) by NYKevin (subscriber, #129325) [Link]

I think we agree: Language-specific package managers are essential for local development, and should not go away.

What I'm saying is that, assuming you eventually want your software installed on an end user's computer (i.e. your end users are not a subset of your developers), then you have to play the usual packaging game, or provide tarballs and make it the distro's problem. Telling end users to run pip install foo is not a great idea. Pip is a developer tool. It has a number of sharp edges; for the longest time, received wisdom was (and might still be?) "never run sudo pip, it will break your system."

Lua and Python

Posted Feb 13, 2020 9:38 UTC (Thu) by farnz (subscriber, #17727) [Link]

I think you're underestimating the complexity hidden within DEB and RPM ecosystems; DEB is marginally less forked, because most of the DEB based distros are Debian derivatives, but in both cases, you still need to handle the differences between different distros packaging policy. In other words, the ecosystem is not "RPM and APT"; it's RHEL and Fedora and OpenSUSE and SLES and Mandrake and Mandriva and Debian and Ubuntu and Mint and then some people with NixOS or Arch or Gentoo or Manjaro or other outliers.

Given that you need to do something for macOS and Windows anyway, why not invent a new standard that's uniform for your language users? After all, there are at least 4 major RPM distros to choose from (treating CentOS as RHEL), at least 3 major DEB distros, and a long tail of oddballs, and having one standard option for $language (and letting Linux distros handle the conversion from $language to $distro packaging in any way they see fit) is less work than trying to care about just the major Linux packaging ecosystems, ignoring macOS/Windows/Android/iOS/WASM etc.

Lua and Python

Posted Feb 12, 2020 19:27 UTC (Wed) by koh (subscriber, #101482) [Link] (5 responses)

I've gone through the trouble of checking every language-specific PM in the thread starter's list. None have been created by whoever conceived the respective language. It's therefore not been a case of
> [..] they probably wrote it in their own language which you don't understand or like (which is why you made your own language) [..]

That reason I can somewhat relate to:
> [..] and because you don't want your language to be weighed down by a dependency on someone else's language [..]
However, if that reason sticks it is just another instance of the not-invented-here syndrome which forcing upon users and developers alike is not productive.

Instead, that is exactly my point:
> [..] they probably didn't design it to be generic enough [..]
By definition, distributions' PMs are generic enough for a distribution. How much more generality than the amount of different (in)compatibilities these deal with is required by any language-specific PM? Is any answer to this question really related to a specific programming language - and, if not,- doesn't that mean there is a PM "waiting to get out" which scratches every such generic itch, distros' and language-propagators', alike?

> [..] it's a pain to learn the subtleties of all of them [..]
This is somewhat self-referential, no? There are too many PMs, so let's invent another one - just for us! xkcd comes to mind... The point being, there is an established process for software to get packaged by the major distributions and it is not by creating an own PM.

> [..] a lot of users will complain they can't use the new language features or libraries that you've written, and will want a way to install updates independently of their distro package manager.
Don't these users just want to switch to a rolling-release distribution?

Regarding the non-distribution-like OS(-fragments), there is the question how many of the target audience ("developers"?) actually use just that fragment without anything resembling a PM on that OS, e.g. homebrew, cygwin or the distribution-encompassing Windows Subsystem for Linux.

Lua and Python

Posted Feb 13, 2020 9:34 UTC (Thu) by dvdeug (subscriber, #10998) [Link]

I think you're missing some of the issues. Let's say that Lua decides to use pip. Does the Python developers want to deal with all the minor issues that Lua is going to need? If no, then you're forking pip anyway, which cuts down on the value of using pip. Either way, now you're using pip; but at the head of this argument is someone not interested in bothering to compile stuff on Windows. I don't want to mess with loading an interpreter on Windows and hoping everything automagically works. They're not using the Linux Subsystem for Windows. Even on Linux, there's always cases with version incompatibility between programs. If you write a program for Lua in Lua, you can usually assume that developers using version X of Lua are happy with the version of the package manager that goes along with version X of Lua. In a world with just Linux, I think things would look a lot of different. But Windows just doesn't have a usable package system for stuff like this, and yet is likely to be the operating system of a fair number of developers.

So, yes, it seems inevitable that a package manager for Lua on Linux, Mac and Windows is going to be written in Lua.

Lua and Python

Posted Feb 13, 2020 13:03 UTC (Thu) by excors (subscriber, #95769) [Link] (2 responses)

> None have been created by whoever conceived the respective language.

I meant they're developed by people within the community around that language, not necessarily the specific people who designed the language itself. pip is written in Python, npm is written in JS, gem in Ruby, Maven in Java, Cargo in Rust.

If you're part of the Lua community and are willing to put in the effort to solve package management, it's probably because you love Lua more than any other language. You wouldn't want to extend e.g. pip to support Lua packages, because then you'd be spending the next several years writing code in Python and you'd much rather be writing Lua.

>> [..] and because you don't want your language to be weighed down by a dependency on someone else's language [..]
> However, if that reason sticks it is just another instance of the not-invented-here syndrome which forcing upon users and developers alike is not productive.

I don't think that's NIH: there are real practical costs to having dependencies. I can go to https://www.python.org/downloads/ and download an installer for Windows that includes a fully-functional pip. But if pip was written in e.g. Perl, either the Python installer would have to tell me to go and download a Perl installer first, or else the Python installer would have to include a bundled copy of Perl (which would make the download much larger, and would likely introduce many bugs since Python people aren't experts in how to install Perl). It's much more convenient to minimise dependencies and keep everything in the same language.

(That's partly a consequence of Windows not having a distro package manager that could trivially download Perl as a dependency of Python. But Python can't solve that problem, they have to just work around it.)

> This is somewhat self-referential, no? There are too many PMs, so let's invent another one - just for us!

Yes, I think it's a kind of tragedy of the commons. Each language develops a package manager to serve its own interests, while the proliferation of language-specific package managers harms the wider community. That is unfortunate but it also seems inevitable, especially in volunteer-driven communities where self-interest is the primary motivation for most work.

As a member of the wider community, it's more effective to just nudge them towards a slightly better direction, than to insist they do something completely different which (based on experience of every other language that was once in the same situation) has no chance of happening.

> Regarding the non-distribution-like OS(-fragments), there is the question how many of the target audience ("developers"?) actually use just that fragment without anything resembling a PM on that OS, e.g. homebrew, cygwin or the distribution-encompassing Windows Subsystem for Linux.

pypistats suggests Windows accounts for 10% of some packages (e.g. opencv-python) and 4% of other packages (e.g. scipy), while Darwin is around 2%-4%. (WSL is presumably counted as Linux, and I hope nobody is still using Cygwin nowadays.)

That's somewhat less than I expected, but still seems large enough that a language shouldn't disregard those developers entirely.

WSL is great but has some limitations, e.g. you can't use matplotlib.pyplot.show() to get a nice interactive view of your graph except by faffing around with X forwarding over SSH (which isn't an optimal experience). I don't know if you could develop a game using OpenGL in WSL (apparently X forwarding supports that but I'm not sure about performance or features), but I'm pretty sure you couldn't develop a game with Vulkan. Sometimes developers do need native platform support, so a language shouldn't rely entirely on WSL.

Lua and Python

Posted Feb 23, 2020 11:48 UTC (Sun) by Wol (subscriber, #4433) [Link]

> Yes, I think it's a kind of tragedy of the commons. Each language develops a package manager to serve its own interests, while the proliferation of language-specific package managers harms the wider community. That is unfortunate but it also seems inevitable, especially in volunteer-driven communities where self-interest is the primary motivation for most work.

That's NOT the tragedy of the commons ... it may be a tragedy, but not that one ...

The real tragedy is

Outsiders: "You should use pip as your language manager"
Insiders: "But what we have works!"
Outsiders: "We've fixed pip so it works for you - you have to use it"
Insiders: "But but but ..."
...
...
...
Outsiders: "What the hell went wrong?"
Insiders: <crickets>

The original tragedy of the commons (like so many since) was driven by OUTSIDE forces disrupting a working eco-system. Often but not always driven by a desire of said outsiders to benefit from the perceived productivity of said eco-system.

Cheers,
Wol

Lua and Python

Posted Feb 26, 2020 19:02 UTC (Wed) by Per_Bothner (subscriber, #7375) [Link]

WSL is great but has some limitations, e.g. you can't use matplotlib.pyplot.show() to get a nice interactive view of your graph except by faffing around with X forwarding over SSH

Or you can use DomTerm. Screenshot here (scroll down to "Python language").

Lua and Python

Posted Feb 13, 2020 13:39 UTC (Thu) by NAR (subscriber, #1313) [Link]

The thing is, the language-specific package manager might have more functionality than simply downloading dependencies. For example rebar3 for Erlang and mix for Elixir can generate project templates, build the project, run tests, create releases, publish to package repository, etc. Both are also extensible - and as they cater for Erlang and Elixir developers, the extensions can be written in Erlang and Elixir. Adding this kind of functionality to e.g. pip would be huge and complicated work, not to mention the dependency on Python for Erlang or Elixir development. So I don't really see the market for a language-agnostic package manager, because it would have less features than the already existing package managers. For projects written in a mixture of languages, there's always make to add the necessary glue.

Lua and Python

Posted Feb 13, 2020 7:48 UTC (Thu) by samlh (subscriber, #56788) [Link]

System package managers are for users.
Language package managers are for developers.

Despite wearing both hats, I want very different things when I'm working on a project, and when I just want to run a thing.

The best language package managers are tightly integrated with the compilation system, so that everything you need to resolve and download dependencies, configure, build, test, and document your code is one simple command away.

This level of tight integration is not something that can be easily abstracted to work for multiple langauges.

See this article for a nice overview of the considerations you need to keep in mind when designing a language package manager:
https://medium.com/@sdboyer/so-you-want-to-write-a-packag...

Lua and Python

Posted Feb 13, 2020 19:30 UTC (Thu) by jschrod (subscriber, #1646) [Link]

> Why is there a need to reinvent package management for every language?
> The basic task is the same as the distribution's PM.

No. Language package managers integrate with development environments, distribution PMs don't do that. E.g., they are often used to provide templates for new projects in a way that is "best practice" for the given language or project. Maven integrates with Eclipse or other Java IDEs, RPM or DEB doesn't. Besides, I don't want to install every development library system-wide just because I need it for some experimentation. I want to try it out in my development environment without impacting my system setup.

Incidentially, usage of distribution PMs for development lead to preferred usage of stuff like Docker, which IMHO is a problem in itself. (Especially, when the developer's version is thrown over to ops and put into production; something that seems to happen 99.99% of the time. Horrible, to deploy such setups without a proper devops setup that cares for dependency and security management.) But, I digress. Coming back to your next question:

> What is so language-specific?

Please note, that this is a different question. A development PM is different from a distribution PM because it is used in development, not because it is specific for a language.

On the question why different language development communities chose to implement their own PMs, others have already given adequate answers: It's not a technical issue, but usually an organizational, deployment, and portability issue.

Lua and Python

Posted Feb 19, 2020 22:32 UTC (Wed) by jo42 (subscriber, #59640) [Link] (1 responses)

Today I found Conda: »Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN, and more.

Conda is an open source package management system and environment management system that runs on Windows, macOS and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was created for Python programs, but it can package and distribute software for any language.«

Lua and Python

Posted Feb 20, 2020 14:37 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

My issues with Conda are similar to those many have with distro systems: either you're all the way in or you're a leaf. There are some systems where you get libraries from the vendor (higher than libc) that "have" to be used on that platform. There's work on getting Conda to accept those, but it's like trying to get RPM to accept your custom installed X11 (that isn't controlled by RPM itself).

It's a nice idea, but I see it mainly as a(nother) distribution mechanism rather than a development base or any kind of omni-solution.

Lua and Python

Posted Feb 12, 2020 18:11 UTC (Wed) by tzafrir (subscriber, #11501) [Link]

The article did mention LuaRocks.

Lua and Python

Posted Feb 13, 2020 4:46 UTC (Thu) by Kamilion (subscriber, #42576) [Link] (3 responses)

Huh. Generally the place I encounter lua the most is game engines. And to be quite honest: Nobody wants 'batteries' there, at worst they're a security vulnerability waiting to happen (Starbound's lua interpreter can't even do file I/O for example, it has to go through the C++ engine code to do so) and at best they're just excess baggage if they're not already incredibly performant. But for "simple" AI routines, NPC dialog trees, and other tasks, what good is amiga AIFF audio support? (one of the python stdlib batteries scheduled to be dropped)

Factorio's lua support for modding exposes a lot of helper functions that will never exist in any other lua interpreter.

Prison Architect's engine blacklists network functions from running. Teleglitch and TOME/Tales of Maj'Eal both expose engine state to the interpreter using structs incompatible with the lua-struct type identifiers.

Fivem's lua injector for Grand Theft Auto 5 allows completely arbitrary network and file I/O and there's been at least three instances of malware being spread.

Heck, I even have a repo stuffing lua5.1 into the .NET process of Terraria and enumerating and exposing all of the internal functions.
https://github.com/kamilion/WraithModRevival/tree/master/...
It even patches the game to create and enable a quake-like console when you press `/~.

NONE of the lua scripts above are compatible with each-other in any way. Every single one of them relies on implementation specific bridge functions that do not exist in the others.

And honestly? Lua's really the only interpreter that does things this way.
S'why I constantly have to tell all the tweenaged gamers on discord utilizing my server hosting, "There's no books to learn lua for this task. All you can do is experiment."

For non-gaming uses, look at OpenRESTY, embedding lua into the nginx service. Quite nice if you don't want to run uwsgi+python for a simple microservice endpoint.

Lua and Python

Posted Feb 13, 2020 11:03 UTC (Thu) by excors (subscriber, #95769) [Link] (2 responses)

As another gaming example, Civilization 4 used Python, but Civ 5 switched to Lua. Presumably their experience with Python was less positive than they hoped for. I don't see any official comments explaining why they switched, apart from a complaint about Boost.Python's effect on C++ build times (https://games.slashdot.org/story/05/10/27/059220/answers-...), but it does suggest Lua is the better choice for this use case.

Personally I'd prefer that Lua remain tightly focused on that kind of embedded use, rather than aiming to be a general-purpose language that can compete with Python, because Python already exists and is alright, and it seems inevitable that a widening of scope would result in compromises that make Lua less good in the embedded niche where there is currently no good alternative.

Lua and Python

Posted Feb 13, 2020 18:29 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Lua interpreter is single-threaded, but it doesn't have any global state. So you can easily run it concurrently on multiple threads, even with LuaJIT if needed. CPython is always single-threaded and global.

Lua and Python

Posted Feb 15, 2020 11:21 UTC (Sat) by togga (subscriber, #53103) [Link]

Well. As numerous comments suggest, with a packet manager you can have it both way. And while Python has drifted towards the moon in bloat/features and has dropped the embedded case (both embedding and communicating with embedded devices) I'm thrilled to see a scripting language adding batteries while keeping everything lean and mean.

Ditch the dynamic library

Posted Feb 23, 2020 11:37 UTC (Sun) by Wol (subscriber, #4433) [Link]

Bit late responding, I know, but it sounds like the problem is not a standard library per se, but everybody is so entrenched in the concept of dynamic linking that they forget Lua does not appear to have the problem it was designed to solve ...

In the days when disk was short and memory expensive, the dll meant that you only needed one copy of the library. Even if each program only used 10% of the library, once you got to 11 programs you were quids in. Now dynamic libraries are getting bigger and bigger, and each individual program uses less and less of what's available, so you need more and more programs to make the savings stack up.

Lua sounds like you are often only running one or two programs, and the overhead of a dynamic library is horrendous. Go back to static-style linking - the *linker* only picks out of the library the functions that are needed, and copies them into the program.

That way, you can have a large standard library, with lots of standard functionality, that does NOT bloat all your small programs. Yes it may mean more copies of the library all over the place, but that's only a bad thing in the wrong place - which is not here.

Cheers,
Wol

Lua and Python

Posted Jan 15, 2021 22:35 UTC (Fri) by RobertX (guest, #138591) [Link] (1 responses)

They should do what Tcl has done. You get Tcl and that comes with stuff and then you can add tcllib which adds a lot more libraries. No reason Lua couldn't stay slim and small and if you needed the "batteries" download the lualib package to get the extended standard library stuff.

Lua and Python

Posted Jan 25, 2021 9:49 UTC (Mon) by ceplm (subscriber, #41334) [Link]

The problem is authority: when there is one standard library blessed by BDFL, everybody grumbles but in the end collaborates on it and uses it (Python, Ruby, etc.). There is no way whole community using one language would agree on one library to be the standard. In case of Lua, where the community is mostly excluded from participating in the development of language, and where Roberto didn’t designate any library to be the standard (actually, Lua is not that bad as NodeJS ‥ there is at least some functionality included in the standard interpreter), there is no way the community would agree on any one library to be the standard one.