-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lua 5.3 or luajit support needed #5205
Comments
I think by now it is probably okay to depend on Lua 5.3 while dropping compatibility with older versions of the language. |
@lachs0r unfortunately mpv with lua 5.3 doesn't actually work yet otherwise that's exactly what we would do :) |
Luajit is already used by various people and should work (I'm pretty sure I build my Windows builds with it). 5.2 to 5.3 upgrade on the API side is not a simple task, so if you have luajit packaged you might just do that on your side. If someone gets bored they can of course enable upstream 5.3 to work as well. edit: I don't have any MacOS build VMs around right now so I can't say much about LuaJIT + mpv on MacOS. Just that in general it seems to work. |
Last time I tried: luajit doesn't work with mpv on OSX. wm4 once told me on IRC and I tried to confirm it, successfully: #1110 (comment) I didn't try to figure out the problem back then as I assumed there was a reason it is (still) broken although known. A switch to 5.3 will also (most likely?) require modifications to all built-in lua scripts. And then we are still left with dozens of potentially incompatible user scripts: https://github.com/mpv-player/mpv/wiki/User-Scripts |
By default, upstream lua (and Homebrew) build 5.3 with the 5.2 compat option, so I don't think breaking the user scripts is an issue unless they're doing 5.1-only things. |
This appears to still be true. Not sure why that would be macOS specific? With luajit
vs. with lua@5.1
|
Lu 5.1, 5.2, and 5.3 are different languages. You can compare it to Python2 vs. 3, except with worse compatibility. 5.3 in particular adds an integer data type, which requires major changes both to API user and Lua scripts. The often cited 5.2 compatibility in 5.3 doesn't seem to exist. There are some compatibility flags, which control the visibility of new and/or deprecated symbols, and a flag for something metamethod related. There is nothing that makes 5.3 run unmodified 5.2 scripts, nor anything that removes the need for major changes in API user code. So this compatibility mode is a red-herring from what I can tell. But even if it weren't, we couldn't rely that everyone who builds mpv builds Lua with this compatibility enabled. It could even be that such a compatibility mode (which would be a hack from the Lua creator's point of view) would not conflict with legitimate Lua 5.3 API users and scripts, which would further decrease the chance of support. Moreover, LuaJIT does not support 5.3 (apparently because the author dislikes the language changes). Switching to Lua 5.3 would mean dropping LuaJIT support for good. Even if you ignore the script compatibility issue, supporting both 5.1/5.2/LuaJIT and Lua 5.3 would be a big mess. There is a large number of user scripts: https://github.com/mpv-player/mpv/wiki/User-Scripts Certainly we don't want to brick them over night, but from what I can tell, the newly introduced integer type could cause random runtime failures because the scripts do not expect it. I don't know how bad it actually would be (there's a small chance all issues can actually be avoided), but at this point it just seems not worth supporting 5.3. In fact I have yet to hear a good reason to support 5.3 (or even 5.2), other than that it's "newer". But be reminded again, 5.3 is not just a compatible maintenance release over 5.2 (as the version number might suggest), but a new language based on 5.2. There is a bit of history about this issue. There is a small FAQ entry, at least 2 rejected PRs for adding 5.3 support, and probably a bunch of closed issues. We won't change just because an overzealous maintainer doesn't understand the issue and demands that we do things that were discussed and rejected years ago. Anyway, as it stands, there are three choices:
(LuaJIT does support OSX, but because of some OSX idiocy (typical Apple), a special build flag is needed: http://luajit.org/install.html (search for image_base). I think it's actually in pkg-config, but waf filters it out for god knows what awful reasons.) |
Just so we're all clear: that person is me and I'm not demanding anything. |
Well, making an issue that 5.3 support is "needed" sure sounds demanding to me. mpv will probably require 5.1/5.2 for a long time, so dropping support for those in homebrew also implicitly puts pressure on us to move to a different version of Lua (and, if removed knowingly, certainly implies a strong demand). It seems you're also suggesting LuaJIT. That is merely a third party implementation of Lua 5.1, with improvements. I don't know if it's as portable as official Lua, but there sometimes are problems associated with it doing JIT (such as the OSX runtime problem). |
@wm4 what title would you suggest instead? I'm happy to change it. |
5.3 has been the current upstream version of Lua for nearly three years now, there hasn't been a new 5.2.x release in as much time, and there hasn't been a 5.1.x release in nearly six years. By contrast, LuaJIT 2.0.5 and 2.1.0-beta3 were released in May of this year. So, yes, if Lua > 5.2 is going to be in the "never" column, looking toward a language implementation that's acceptable and currently maintained seems reasonable to me. That said, according to our analytics for the last 30 days Install events: Install-on-request events: So I would recommend trying to stay on the mainline Lua train if possible in terms of what users will likely be wanting to use over the longer term. |
As I've been trying to tell you, 5.2 and 5.3 are different, incompatible (even if similar) languages. On some level that's like trying to argue that we should switch to JavaScript because JS is more actively developed. I expect that users most likely want working user scripts. Also, nobody ever asked us to support Lua 5.3 because they wanted features that are only in 5.3. (Actually, it appears Lua is mostly hated by those who try to write scripts, but for some reason it seems it's still preferred over mpv's JS scripting support.) Even if 5.3 and 5.2 are very similar, the change will brick user scripts, and that's not really acceptable. Note that I haven't really tried to quantify how much will break, so you're welcome to try to update mpv to use Lua 5.3 and demonstrate that it doesn't break most scripts (Lua 5.3 compiled without compat options, see my previous post why we wouldn't want those). |
The upstream default configuration and binaries are compiled with the 5.2 compat option, so it's rather peculiar to take that the position that only 5.3 without compat is worth considering. |
Which compat options are you specifically talking about? There is LUA_COMPAT_5_2 (which is indeed enabled by default), but affects only very little of the language or the API. The define mentioned above does the following:
In particular, the compat option does not:
In addition, the manual states that these compat options will be removed in the future. PRs which tried to support are #1516 and #1914. As you see, they both required changes to the C code and our internal Lua code. (Keep in mind that the OSC is basically a user script, just shipped as part of mpv. This indicates that user scripts would probably need updates as well to work with 5.3.) All this is actually the same for 5.1 vs. 5.2: some scripts work only on one of those versions, and mpv's C code has some horrible ifdeffery to attempt to support both. But I suspect the situation is way worse in 5.3 because of the integer change, and it would mean we have to support/test 3 Lua versions, which would mean reconfiguring/recompiling 3 times on every meaningful change to the Lua wrapper, or our internal Lua scripts. (I guess we could drop Lua 5.2 support though, which would make it a bit easier again.) So stop pretending this is easy, just because you don't want to maintain a 5.1 package. At this point you wasted more of our time than it would have been effort to maintain such a package anyway. In any case, anyone is free to analyze the impact of Lua 5.3 support further (both in mpv's code and user scripts), but I still think this integer type introduction poses inherent compatibility problems that are not easy to fix. There are still 2 fixed outcomes:
Doesn't seem very attractive. Just to support some maintainers a few minutes of work every other month. |
This. |
Well, it doesn't do anything you claimed it does. If you would read my posts... |
@wm4 I've read your posts. Try to scale back the hostility a few notches. |
Well, it would be nice if you replied to any of the technical points, instead of focusing on perceived hostility and making an issue out of that. |
I'm not pretending anything is easy so try not to stick words in my mouth. I am telling you that we won't support a lua@5.1 in perpetuity.
It's not about the effort to maintain the package. It's about carrying software that has no upstream support, which is the case if there isn't an actively maintained separate 5.1.x release line. It sounds like the best option at least for the medium term will be to find a way to get the formula working properly using |
It looks like mpv works with LuaJIT 2.1.0-beta3 built with
Not sure yet why So it does seem likely LuaJIT 2.0.5, which doesn't have the |
@wm4 Is there a reason to prefer Lua 5.1.5 over LuaJIT, or LuaJIT over Lua 5.1.5, from your perspective in terms of what mpv should use? |
Just for completeness for anyone unaware: Lua 5.3 handles numbers differently, which breaks compatibility with scripts. In all previous versions of Lua, numbers were always floats. A number such as "103.0" was printed as "103" and treated as an integer everywhere. A number such as "103.1" was printed as a float. In 5.3, an integer is always an integer and a float is always a float. A float of "103" is printed as "103.0". And in a player where numbers are used for on-screen display formatting, assumptions about decimals (103 vs 103.0) were never a problem before but could be a big problem now. It changes everything about numbers and requires analyzing every Lua script to make sure they're going to survive the new number handling (for example: some core scripts in mpv require rewrites to round numbers properly in Lua 5.3). Another big problem with supporting 5.3 (even with script-side "compat flags") is that some people/distros may compile 5.3 without compat flags, and those flags are going to vanish in a future 5.3 release, and other distros may not even bundle Lua 5.3 at all. Supporting another language (5.3) isn't viable. As for solving this via LuaJIT: Seems okay unless there are bugs associated with that VM together with mpv (no idea, have personally only used mpv with Lua 5.1). @wm4 mentioned some OS X issues with LuaJIT's JIT above... |
In theory, you could write code that doesn't care whether a number is integer or float, but it seems pretty easy to run into conflicts anyway (this affects both in C using the Lua API, and Lua code). @ilovezfs not really. LuaJIT provides the same API and language as Lua 5.1, so it doesn't matter for us. LuaJIT might have a different set of obscure bugs. |
We can't ignore the issue forever.
Not all downstreams are going to be willing to ship old versions of Lua forever. Eventually Lua 5.4 is going to come out, then Lua 5.5, then maybe Lua 6.0... the situation is only going to get worse. Even if the change is difficult it's still something we're going to have to face. Maybe we can try to support both for a while, but eventually we're almost certainly just going to have to let user scripts break. |
The problem is that lua maintainers, as far as I know, are willing to break backward compatibility of existing scripts, which makes it hard to depend on them in the long term. If it was only the embedding API then it's probably worth the effort to move to new version, but if a new version can break scripts then it's a really big problem. With javascript there's standardization, and it's well defined how new versions should behave with regards to older scripts - typically just fine. Of course, js is not perfect either, because well-backed implementations are far from trivial to use (v8 etc), and smaller implementations have the usual issues of projects without many active developers. It's a real problem overall. |
We can petetion the maintainers to avoid breaking backwards compatability in the future. We can push for language standardization. Let's express our concerns with them and try to work together instead of in a vaccum. |
@avih Yeah. Lua is one of the worst languages in the world from a maintenance perspective. They need to learn semantic versioning and respect backwards compatibility. In any other library, 5.1 and 5.2 and 5.3 imply minor versions, but in Lua they are all incompatible with each other. What's the point of having version numbers if they have zero meaning to Lua? They should just name it Regarding JavaScript: Until today, I didn't even know that mpv had JS support. It uses "MuJS", a tiny single-file, super basic ES5 implementation. It's unfortunately lacking modern ES6 features. But still, JavaScript is lightyears ahead of Lua. For example, Lua doesn't have a built-in string split function (even though splitting and joining strings is one of the most common operations in any program). Instead, it has a "pattern matcher" (gmatch) which is only able to match the text chunks themselves (as opposed to their delimiters). How can anyone take such a language seriously? Lua is only prevalent because its engine is so lightweight and easy to integrate. But as a language it's ridiculously bad. Regarding MuJS: There are runtime |
@heelsleeh you completely failed to understand the issue. Upstream maintainers (like for homebrew) don't want to maintain "old" lua 5.1, which create an issue for users, because mpv doesn't want to break existing user scripts by upgrading to a newer versions, but homebrew (and others) don't want to keep supporting lua 5.1 forever. It's a real issue, which stems from the fact that lua maintainers break compatibility with their new versions. This is beyond the control of both mpv developers and upstream maintainers, and yet both suffer from it. |
Homebrew looks like it'll be a continual complainer here, at some point they'll "upgrade" 5.3 to 5.4 or 5.x, ect. |
@mc4man I'm not sure there is any single right way to manage stuff like this, because different projects have different values and conventions. From my perspective (and I don't represent homebrew, or anyone else), the job of a package manager is to be aware of these problems and try to minimize their impact as much as possible for end users. That includes having a dialog with project operators. I found an old Macports thread about this general problem with Lua. Personally I agree 100% with Ryan in comment#20 where he advocates for distinct version-locked lua ports, which are perfectly fine existing along side a moving-target lua package. Today Macports still offers both "lua", and also "lua50", "lua51", "lua52". I think that's a reasonable compromise, as long as the cost of maintaining the older lua packages is reasonable. If those older Lua versions are not maintained by the lua project, eventually they will become impractical to maintain, because package managers are probably not willing to take unlimited measures to keep them working in the face of all the standard forces (like entropy). Software can become obsolete for a variety of reasons that do not apply evenly across the entire population. That's not a bug, that's a feature :) The ability to use different versions of software includes the responsibility of managing the versions of software being used (this applies to users, developers, and package managers). I'm not saying everybody should be hamstrung by the lowest common denominator, but rather that some form of compromise should be reached. So anyway... I agree that brew would do well to switch mpv to a version-locked package of lua, because mpv gets no direct benefit from upgrading, and some significant downside. However, even though you may not care about the cost for homebrew of maintaining old lua ports, I think it would be helpful to acknowledge that this cost does exist, and may grow in the future. After all, lots of people get to mpv through homebrew, so in theory there is some shared interest here. With all that in mind, if there are low-cost practices for mpv script authors that can make a possible future Lua upgrade easier, and those practices can take place slowly over time, I think it's worthwhile. I'll do some investigation. |
The point is: who knows? If lua 5.4 breaks compatibility again then we're back at square 1, so what's the point of such effort if it has to be made over and over? If the lua maintainers declare that they will make a strong and meaningful effort to keep scripts compatibility (and highly preferably embedding API compatibility) to some lua version, then sure, it's worth the effort and breakage, but if not, then IMO it's just not worth the change, and lua 5.1/5.2 will remain the supported version for as long as possible, and once it's not possible, who knows. |
Could just send a message, drop support for Lua as it diverges and keep Luajit. |
@Teathief Agreed. Lua has no respect for backwards compatibility, which is fine for static linking inside game engines but not in big apps that dynamically link to random system-versions of Lua. LuaJIT is a much safer bet for long-term reliability. |
Thanks to @wiiaboo , here's the exact issue:
So the only solution I have so far is to manually set
This ends up with working luajit in mpv, confirmed working both with macports luajit and with my own luajit git-master which I built myself, and I'd guess it will work with homebrew's luajit too. |
Considering you're using Devel + GC64, it could be the cause of the bug. I suggest that you try my method with the release version and the manual LDFLAGS setup. Devel is luajit 2.1 (beta 3 currently), GC64 is not enabled by default at the Makefile or homebrew, and considering this relatively recent comment:
and the (relatively slow) pace of luajit development, I'd say it could be it. |
|
To add a non-technical perspective, having only just discovered mpv, I went to install it but then stopped when I saw it depended on an old version of Lua. In my experience, outdated dependencies is usually a sign of a dying project. I found this thread out of curiosity but I'd guess others just see lua@5.1 and don't install. |
Yes, person who no longer works on mpv, please change the fabric of the universe to make sure that bigger number not bigger so user looking at dependencies not confused! Me think too different for look at repository to see if project alive, me now off to charge AirPods. Mmmm crayons. |
@CounterPillow I think you're missing the point. A layperson isn't going to look at the repository. But I'll let you get back to your crayons 😉 |
A layperson isn't going to look at dependencies, comparing them to the latest version. |
I'm a layperson. I don't know the difference between Lua 5.1 and 5.3. I don't know Lua. I just know that mpv wanted to install the old version, which made me wrinkle my nose. |
Choose one:
This is a tough one. |
I guess you assume most people ought to prefer the first option, but I think this is misguided. If the layperson passes over a project at the installation stage, it doesn't really matter what's going to break afterwards. I'm not suggesting the maintainers should make appeasing my laypeople brethren and I their top priority, I was just adding a perspective that those smarter than me perhaps wouldn't have considered, i.e. given my naivety concerning different versions of Lua, I was hesitant to install an old version because I don't know if that would cause me problems. |
I don't think multiple Lua versions replace each other. If they do that's a deliberate choice on the packager's side. For example, Fedora (and I think Debian f.ex.) still have Lua 5.1 as Also if you don't like to install older official Lua or for some reason or your packaging system doesn't support installing two of them next to each other, then LuaJIT is another alternative, which is an alternative implementation of the language as it was with Lua 5.1. I have been using it with my Windows builds for quite a while now. I think at some point there was a patch to add Lua 5.3 support, but I don't think anyone even had the time to test if the scripts included in mpv (including the on-screen controls and youtube-dl integration). Even if we ignore all external scripts, we would at the very least have to test those included with the player itself and add a warning to users that if you choose to build with 5.3 it can cause breakage with scripts. |
There's also a FAQ entry: https://github.com/mpv-player/mpv/wiki/FAQ#Why_does_mpv_not_support_Lua_53 In summary, I don't think mpv will ever switch away from Lua 5.2, even if it's considered "outdated" by some. Unless a newer Lua version (Lua 5.4 etc.) restores some compatibility, or someone can demonstrate a way to switch to 5.3 without breaking user scripts. Both pretty unlikely. |
luajit works, lua5.3 support won't come for reasons mentioned. colsing. |
Lua 5.2 is dead for more than 6 years by now, and even Lua 5.3 is EOL. Current major release is 5.4.3. I think most of popular user scripts would be ported very quickly, and the ones that won't just isn't worth much to be worried about. Users can very happily live with older version of MPV supporting those scripts for some time. And I dare to say most users aren't even aware of Lua scripting capabilites of MPV. TL;DR: I don't see any valid reason not to support current language version besides developers' laziness; that's a valid reason IMO, but should be stated clearly and not hidden behind some very shaky excuses. |
ok Jasuf |
I have upgraded lua in Homebrew from 5.2 to 5.3 in Homebrew/homebrew-core#21300. This broke lua support, so mpv now needs
In Homebrew,
@
formulae for older versions are temporary, so depending onlua@5.1
is not a viable long-term solution for mpv.Some possibilities for mpv in Homebrew:
depends_on "lua"
depends_on "lua@5.1"
todepends_on "luajit"
(if that works?)The text was updated successfully, but these errors were encountered: