Lua 5.4 update thread

[Meo-Ada Mespotine]

Let's discuss the incompatibilities of the Lua update with existing scripts in here.

What's new in Lua 5.4:
https://lwn.net/Articles/826134/

Incompatibilities with older versions of Lua:
http://www.lua.org/manual/5.4/manual.html#8

[Meo-Ada Mespotine]

math.randomseed has been changed in its arguments, so scripts who use it are affected.

Removing it seems to be working, though.

[Meo-Ada Mespotine]

io.lines returns now multiple return values.
So if you use it within a function that has additional parameters after it, things might break.

For instance something like

myfunction(io.lines(filename), parameter2, parameter3)

will break, since parameter2 and 3 will be filled with the return values of io.lines().

The fix for these seems to wrap io.lines into parenthesis:

myfunction((io.lines(filename)) , parameter2, parameter3)

though I haven't tested that.

[Sexan]

randomseed now expects integer

https://forum.cockos.com/showpost.ph...5&postcount=52

math.randomseed(math.floor(x))

would fix it, but would the behavior change in that case?

[X-Raym]

@Sexan
As far as I understood, randomseed is mostly not necessary anymore.

[Sexan]

How to keep compatibility with prior 5.4 scripts?

[X-Raym]

I quickly inspected and tested current usage of randomseed in ReaScript world (11000 files):
709 matches across 358 files


But not all breaks:



This is what I found:

> math.randomseed()
{1691320194, 99641400}
> math.randomseed(1)
{1, 0}
> math.randomseed(reaper.time_precise())
bad argument #1 to 'randomseed' (number has no integer representation)
> math.randomseed( os.time() )
{1691320229, 0}
> math.randomseed( os.clock() )
bad argument #1 to 'randomseed' (number has no integer representation)
> math.randomseed(math.floor(reaper.time_precise()*1000)) 
{407012496, 0}
> math.randomseed(reaper.time_precise()*os.time()/1e3)
bad argument #1 to 'randomseed' (number has no integer representation)

So what breaks:

• math.randomseed(os.time()) format for eg is about 20 files.
• math.randomseed(os.clock()) is 4 files
• math.randomseed(reaper.time_precise()*os.time()/1e3) is kawa only, 298 files (but I may have this in double )

so according to this data (which may need more careful inspection), it doesnt affects lots of repo, and one of them is more than 99% of the issue.

[cfillion]

Originally Posted by Meo-Ada Mespotine

io.lines returns now multiple return values.
So if you use it within a function that has additional parameters after it, things might break.

For instance something like

myfunction(io.lines(filename), parameter2, parameter3)

will break, since parameter2 and 3 will be filled with the return values of io.lines().

The fix for these seems to wrap io.lines into parenthesis:

myfunction((io.lines(filename)) , parameter2, parameter3)

It's the opposite: the behavior is only changed when io.lines(...) is the last explicitly passed argument.

myfunction(io.lines(filename))

This passes all return values of io.lines as arguments to myfunction. And now there's 4 of them. To pass only one like before:

myfunction((io.lines(filename)))

3.4.12 – Lists of expressions, multiple results, and adjustment

[Meo-Ada Mespotine]

Originally Posted by cfillion

It's the opposite: the behavior is only changed when io.lines(...) is the last explicitly passed argument.

I'm not sure if I understand it really. If it returns now four return values, it shouldn't make a difference as which argument it is passed. Passing as the last one shouldn't be basically like the old behavior but when passing it in an earlier one, it should fill up up to 4 arguments.
At least it's how Lua basically works and my tests indicate this.

So if it really only returns 4 returnvalues when passed as last argument to a function, then this would be odd and impossible to test. Unless it's only passed to ... arguments.

Will test it with the newest pre of Reaper to confirm if I am mistaken, though...

[cfillion]

function myfunction(a, b, c, d)
  print(a, b, c, d)
end

function mylines5_3()
  return 1
end

function mylines5_4()
  return 1, 2, 3, 4
end

myfunction(mylines5_3())     --> 1	nil	nil	nil
myfunction(mylines5_4())     --> 1	2	3	4
myfunction((mylines5_4()))   --> 1	nil	nil	nil

myfunction(mylines5_3(), 42) --> 1	42	nil	nil
myfunction(mylines5_4(), 42) --> 1	42	nil	nil

This feature is not limited to function calls but rather any time an expression list is involved. Variable assignments for instance:

local a, b = mylines5_3()      --> 1, nil
local a, b = mylines5_4()      --> 1, 2
local a, b = (mylines5_4())    --> 1, nil

local a, b = mylines5_3(), nil --> 1, nil
local a, b = mylines5_4(), nil --> 1, nil

[Justin]

Originally Posted by Meo-Ada Mespotine

math.randomseed has been changed in its arguments, so scripts who use it are affected.

Removing it seems to be working, though.

math.randomseed() is supported on both 5.3 and 5.4, the difference is that in Lua 5.4, a parameter must be an integer. So replacing math.randomseed(x) with math.randomseed(math.floor(x)) would work on both.

Having said that, we'll modify Lua 5.4 to make it permissive and thus compatible with 5.3.

[Rodulf]

This is weird. Brian Chi's FX Devices would not work for me. Now, with this change, all of a sudden, it functions perfectly. Go figure.

[X-Raym]

@Rodulf
We have figured 😉

But next pre should make it compatible without modification as Justin proposed ,👍

[akademie]

Originally Posted by Justin

...<snip>
Having said that, we'll modify Lua 5.4 to make it permissive and thus compatible with 5.3.

Ah, right move I think (if it does mean what I think).
Thank you Justin.

With that huge number of ReaScripts already existing in ReaPack repositories it would be very hard, difficult to keep all of them actual, updated, when sometimes original writers are not available

[Justin]

Originally Posted by akademie

Ah, right move I think (if it does mean what I think).
Thank you Justin.

With that huge number of ReaScripts already existing in ReaPack repositories it would be very hard, difficult to keep all of them actual, updated, when sometimes original writers are not available

--- a/sdks/lua/src/lmathlib.c
+++ b/sdks/lua/src/lmathlib.c
@@ -621,7 +621,7 @@ static int math_randomseed (lua_State *L) {
     randseed(L, state);
   }
   else {
-    lua_Integer n1 = luaL_checkinteger(L, 1);
+    lua_Integer n1 = (lua_Integer) luaL_checknumber(L, 1); // cockos mod
     lua_Integer n2 = luaL_optinteger(L, 2, 0);
     setseed(L, state->s, n1, n2);
   }

that does the trick, I think

[daniellumertz]

Originally Posted by Justin

Having said that, we'll modify Lua 5.4 to make it permissive and thus compatible with 5.3.

How a modified version of lua will deal with the workaround we have been using to have Lua modules in reaper?

More info on how it is been done, schwa said(at comment #2) this is the only method we have to use external modules :
https://forums.cockos.com/showthread.php?p=2555363

I'm worried because using the methods we had been using in pre3 with lua 5.4.6 will crash reaper, so with a personalized version of lua wont it come to a dead corner where there isnt anything we can do to have external modules?

[Edgemeal]

This code works fine in v6.81, but errors in v7.0pre3...
attempt to perform bitwise operation on a string value (local 'val')

  local target_column = 3
  local _, val = reaper.BR_Win32_GetPrivateProfileString('REAPER', 'actionflag', '', reaper.GetResourcePath() .. "/reaper.ini")
  if (val & 2) == 2 then target_column = target_column + 1 end

What is the best way going forward? Should I change all scripts?

 if (tonumber(val) & 2) == 2 then

???

[cfillion]

I'd add the tonumber for clarity and robustness even if REAPER's Lua 5.4 gets patched to restore the weird implicit conversion. A bitwise operation on a string makes no sense + since the ini file is external data, without `tonumber() or 0` the script could error out on unexpected values*.

local ACTIONFLAG_XYZ <const> = 2 -- to remember what 2 means years from now

local actionflag = tonumber(select(2, reaper.BR_Win32_GetPrivateProfileString(
  'REAPER', 'actionflag', '', reaper.get_ini_file()))) or 0
if actionflag & ACTIONFLAG_XYZ ~= 0 then -- "not 0" to not type the magic number/constant twice and reduce the risk of mismatch in the future
  -- do something
end

* It depends. The error could be a feature to abort early if an invalid value silently breaking some functionality would be worse.

[MRMJP]

Starting with reaper70pre4_universal I've been getting some errors when running some of my usual scripts.

I would rather not have to update them all. Is there a REAPER fix possible for this so older scripts are not broken?

[cfillion]

Those script will either have to be recompiled for Lua 5.4 (and not be compatible with 5.3 anymore) or be distributed as plain Lua code. Precompiled bytecode is only valid for a specific version.

[MRMJP]

Originally Posted by cfillion

Those script will have to be recompiled for Lua 5.4. Precompiled bytecode is only valid for a specific version.

Totally shitty. This is precisely why I knew that I couldn't invest 100% of my mastering workflow into REAPER. I have SO MANY scripts that were custom made (by somebody else, not me) and that's not even to do 100% of the mastering process. Just part of it, for some projects.

I use REAPER for the analog play/capture work but I'll be damned if I want to trust it for anything more important than that when things can just break like this with no warning or official support.

This might be what finally weans me off of REAPER for good and over to WaveLab 100% for all aspects of the mastering process.

You can totally do that in REAPER, they said. Just script it, they said. Until the scripts break for no good reason.

[cfillion]

I think responsibility and support for a few third-party scripts being distributed in a special fragile way (precompiled bytecode instead of normal source code) lands on the authors of those scripts rather than REAPER's. A public pre-release like this serves as plenty of advance warning allowing for things to be fixed (on all ends) before v7 is finalized.

v7.0preXX is a pre-release for a major update. Some breaking changes and bugs are to be expected: it's not finished + third-party devs only had a few days so far to adjust to the major Lua upgrade. Beta means it should not be used for real work yet.

[MRMJP]

Originally Posted by cfillion

I think responsibility and support for a few third-party scripts being distributed in a special fragile way (precompiled bytecode instead of normal source code) lands on the authors of those scripts rather than REAPER's. A public pre-release like this serves as plenty of advance warning allowing for things to be fixed (on all ends) before v7 is finalized.

v7.0preXX is a pre-release for a major update. Some breaking changes and bugs are to be expected: it's not finished + third-party devs only had a few days so far to adjust to the major Lua upgrade. Beta means it should not be used for real work yet.

I don't disagree, but when I was advocating hard for REAPER to incorporate some mastering specific things to help attract mastering engineers away from DAWs like WaveLab and Sequoia, all I got from REAPER-heads were comments like..."just script it bro", and "anything is possible in REAPER if you just script it bro". Scripts were apparently the solution to any idea I had.

So, I luckily found somebody that's really great at making scripts for some things I do all day every day, but I knew that trying to make REAPER into a mastering DAW via 3rd party scripts was a terrible idea and this is exactly why so I only took it to a certain point which is where I'm at. i use WaveLab to finalize projects since it was built for that and won't decide to stop working one day without support.

At one point I thought REAPER was cool enough and close enough to being an awesome mastering DAW if they added some native things so things like this couldn't happen.

I know that things can break with big updates and this is not an official release, but it's pretty inevitable that this some form of this version will be public sooner or later and I would have come across this issue at that time.

The main reason I'm even into these pre-release versions is I'm hoping for a graphics fix that doesn't seem fixable.

I've had some great years with REAPER but between the graphics mess, occasional freezing upon opening files, and now this scripts fiasco, I think I might be done with REAPER which I'm sure the developers will actually be happy about.

[schwa]

The scripts you are referring to were distributed in such a way that guarantees they cannot work with any other version of Lua. That was a decision made by whoever distributed them. The only way they can continue to work is if we *never* upgrade the version of Lua that is built into REAPER.

I recognize that this change will mess up your workflow, but I don't think it's fair to say "I'll be damned if I want to trust it for anything more important than that when things can just break like this with no warning or official support."

REAPER officially supports Lua. 99% of third-party scripts will continue to work through a Lua upgrade. But we can't officially support third-party scripts that are *designed* to stop working if we upgrade Lua. And this discussion right now is warning that we will eventually upgrade Lua.

[MRMJP]

Originally Posted by schwa

The scripts you are referring to were distributed in such a way that guarantees they cannot work with any other version of Lua. That was a bad decision by whoever distributed them. The only way they can continue to work is if we *never* upgrade the version of Lua that is built into REAPER.

I understand this now but I'm an audio engineer, not a software developer and I prefer to keep it that way. When I started using REAPER, I didn't even know what Lua was let alone there would be decimal point versions of it that could totally break previous decimal point versions of it.

I didn't realize I needed to ask for scripts to be made that will never break in the future. I thought the scripts were done in such a nerdy and simple code-based way that they would be more future-proof than this.

But again, it underlines my entire point that I should have known better than to build too much of a workflow around 3rd party scripts that can break at anytime, regardless of who is at fault.

I'm just glad didn't take it any further as other suggested I do to get it working more like an actual mastering DAW such as WaveLab or Sequoia.

[daniellumertz]

Update: Sexan and me manage to rebuild the mavriq lua sockets extension with lua 5.4.6 for REAPER v7 and it is working fine!!!! It is also more future-proof for windows now!

if anyone already want to try out the files are here:

https://github.com/daniellumertz/mavriq-lua-sockets-testingnewlua/actions/runs/5790768779

It also seems to work nicelly with the new modfied lua version, I was scared that it did some kind of crash, will keep an eye on that and try to test heavily in the next days!

I will make a proper update at my Socket demo when v7 releases!

[X-Raym]

@MRMJP
What we are trying here is to test if this lua update worths it.


Not all scripts have troubles with the update. But all benefit from significant performance boost.


Those with randomseed had issues, but it will be fixed as mentioned above, and those precompiled apparently (so only 2 so far, those from your report).
We cannot hold back the whole scripting community to a 2015 version of Lua for 2 scripts.


These devs are still active though you can contact them. Fixing this is easy on there side and it well be done for sure once it will be reported.



And if you find other devs which need to be contacted let us know, as we are trying to measure what impact this update can have, and let them know too. But please focus on the technical aspect of the report, we need to keep this thread balanced between reports and coders discussion about the update.



@daniellumertz
Excellent news !

[akademie]

^^^^^^^^^^ Thanks for your description of the situation with LUA update X-Raym, appreciated

[X-Raym]

Originally Posted by X-Raym

These devs are still active though you can contact them. Fixing this is easy on there side and it well be done for sure once it will be reported.

https://forum.cockos.com/showpost.ph...ostcount=2974:

Originally Posted by Heda

Thanks. No future problem. Next update will work with v7

Heda's scripts issues can be considered cleared, at least for v7 release time. in time This is why reports have to be made to scripters.



The other script mentioned with binary issue listed is from Neutronic. Reports have to be send to him.



@akademie
thanks for your encouragements

[Sexan]

Binary version is I guess for obfuscation reasons since there is no difference in performance?

[MRMJP]

Originally Posted by X-Raym

We cannot hold back the whole scripting community to a 2015 version of Lua for 2 scripts.

FWIW, it’s not just two scripts. These are just two scripts I happened to try within about a 2 minute window.

If it were just two scripts I wouldn’t be wasting time here over them, I’d just have them fixed.

It’s going to be MANY scripts, and I do not have the time right now to test every script I have and see what broke. And I’m just one user out of presumably thousands that have some or many scripts.

[Sexan]

TBF it would happened if it was a bugfix update (lua 5.3.0 -> lua 5.3.1). Its literally unfixable since the approach of distributing that script is "wrong". He (author) may have his own reasons to do it (obfuscation is my guess) but only thing you can do is ask him to recompile.

Anyway I'm not aware of any binary distribution on ReaPack ?

[X-Raym]

@MRMJP
The number of scripts is in fact irrelevant.



What matter is the number of scripters implied.


Scripters can fix their scripts in batch. Often, a single search replace on a whole repo is needed. Heda will fix his by time of v7.0 pre release. Neutronik have to be contacted.


For now, only v6.x is suitable for production.


@Sexan
Yes, it is often used for premium stuffs, like the two shown by MRMJP. That is why it is expected that these scripters actually do maintenance of their scripts (paid products), if they are still in business, considering that so far no critical unfixable issues have been found. But their users have to report mostly to them, and then here if it is an issue or dev for which we dont already know there is an issue.

[MRMJP]

Originally Posted by X-Raym

@MRMJP
The number of scripts is in fact irrelevant.

The number of scripts is in fact relevant when you tried to make me feel silly for complaining about just two scripts breaking. Those are just the two I found in just a couple minutes of doing a random task in REAPER. I imagine it will be much more when I’m actually trying to get some work done.

Again, I am only using these latest pre-release versions to help fix another REAPER bug but I’m giving up on that, and probably giving up on REAPER in general for good over the next few weeks.

The last year or two I’ve been sticking to only official releases, but a few bugs and things have gotten me back into pre-release world.

If I wouldn’t have been trying the pre-release versions, I’d probably be in this same situation, just a few weeks or months later.

Maybe they can add a legacy Lua option so old scripts don’t break. The scripts I use are pretty simple I think and I don’t see how a performance increase is going to be so great that it’s worth this headache.


REAPER is a cool DAW but only if you have free time to play around and make it usable. I’ve reached my limit here with this.

[cfillion]

If there's a next time: request that script authors send you the full script source code for future-proofing. If opening the script files in a text editor doesn't show human-readable words it means you're dependent on those authors to update them when they break again in the future.

Technically a precompiled script should load a tad faster (unlikely to be enough to notice in most cases). However once loaded they won't run any faster than a regular source code release would. The only differences that matter are 1) it's harder (not impossible) to see what they do 2) it will break whenever Lua is updated in REAPER.

If you have a critical precompiled script from some author that isn't around to update it anymore: pre-compilation can be reversed to restore a broken script. It's a lot more work than just having the original author send the real script, but it can be done.

[MRMJP]

Originally Posted by cfillion

Technically a precompiled script should load a tad faster (unlikely to be enough to notice). However once loaded they won't run any faster than a regular source code release would.

So what you’re saying is that REAPER is breaking a bunch of scripts so they can load a tad (but unnoticeable) bit faster?

I would ask that this unnoticeable option be a feature or option in settings for those who want to use it, but this sounds like a lot of unnecessary work for an import vent that will be barely noticeable at best.

[X-Raym]

@MRMJP
What I'm trying is definitely not to make you feel silly but guiding your report effort toward the shortest path to resolution, aka contacting the author, while reassuring other users who havent already make their mind about REAPER being bad, that it is false to claims all scripts will become unusable. Some are broken, and lots have already been fixed since the opening of this thread, either by Cockos, or by scripters themselves. This should implied the devs you mentioned, so we should at least saw a bit of appreciation, even minimal, toward our own effort to document the compatibilty errors.



Individual complains about REAPER as a DAW of choice in general can be meaningful in lot of stituations, it is out of place in this very precise thread. We need to focus on code issues here.



However, knowing which scripters have issues is relevant information so thank you for that.

'm still not sure neutronik have be contacted though.

I don’t see how a performance increase is going to be so great that it’s worth this headache.

It's estimated about a 40% increase in all metrics so that is not neglitable. Lua 5.4.3 is 2023, the actual version is 2015. We have to move foward (or at least try and see how issues down the way can be manageable). But we cannot know if we dont try.

[cfillion]

Originally Posted by MRMJP

So what you’re saying is that REAPER is breaking a bunch of scripts so they can load a tad (but unnoticeable) bit faster?

No, I meant that there are no good reasons for scripts to be shared in precompiled form like that.

[MRMJP]

Why can there be an option within REAPER that uses 5.3 (legacy) or 5.4 Lua language?

Similar to how you can open apps in Rosetta Mode or Apple Silicon Mode.

Without that, there is no point in adding any other new features to these pre-releases because at least speaking for myself, I can't test any further updates until I get all my scripts working again which may never happen.

7.0pre2 is likely the last version of REAPER I'll ever be using.

[X-Raym]

Originally Posted by cfillion

If you have a critical precompiled script from some author that isn't around to update it anymore:

This will be very interesting to know indeed. so @MRMJP, try to report to Neutronik, as it seems solved with heda, and let know here if some other authors cannot be reached.