Scripts using Pickle

[tompad]

Hola!

Can anyone hint/hit me with name on a reascript(lua) using Pickle?
I am trying Pickle out and want to look how others are using it.

[X-Raym]

Never heard of that,


if you speak about pickle.lua table serialization module, maybe table.concat native function in Lua 5.3 is enough.


http://lua-users.org/wiki/TableLibraryTutorial

[Lokasenna]

@X-Raym: He needs to serialize a bunch of data for storing in an extstate and reloading.

I believe Sexan uses Pickle - have a look at his Track Versions script, maybe.

[tompad]

Originally Posted by Lokasenna

@X-Raym: He needs to serialize a bunch of data for storing in an extstate and reloading.

Spot on Lokasenna! :-)

I believe Sexan uses Pickle - have a look at his Track Versions script, maybe.

Thanks for tip - will check.

My plan A is to use pickle.lua but I got some errors in pickle code after
changing table.getn (deprecated) to #table so I thought I could check
some others scripts to see how they use pickle.lua.

My plan B is to use this:

 
 reaper.SetProjExtState(0,"ToDoList",1,tostring(GUI.table_list(items)))

Shouldn't that "tostring(GUI.table_list(items))" make a string with table content?

By the way - I watched some other examples with SetProjExtState that used "for k, v in pairs(s) do"
but if I want to save just ONE long string in SetProjExtState do I really need to do a for loop?

[tompad]

I got Pickle running (YEAJJ!!) - but as usual another thing shows up! ;-)

When closing project starting a new project my scriptwindow
stays open!? Shouldn't it close before project is closed?

Edit: I have used the following - none is closing my script window.

local function theEnd ()
  saveToProjExtState()
  --GUI.Window:close()
    gfx.exit()
  --GUI.exit()
  --GUI.quit = true

end


reaper.atexit(theEnd)

Maybe its a Linux-thing?

[Lokasenna]

Originally Posted by tompad

 
 reaper.SetProjExtState(0,"ToDoList",1,tostring(GUI.table_list(items)))

Shouldn't that "tostring(GUI.table_list(items))" make a string with table content?

Yes, GUI.table_list will do that. It also adds pretty indenting though, which might cause trouble.

However, the logic is pretty simple:

local settings = {
  flag1 = true,
  flag2 = false,
  param1 = 4,
  param2 = 5,
  param3 = "tompad"
}

local function settingsToStr()
  local strs = {}
  for k, v in pairs(settings) do
    strs[#strs+1] = k .. "=" .. tostring(v)
  end
  return table.concat(strs, ",")
end

Msg(settingsToStr())
--> flag1=true,param2=5,param1=4 ...etc...

-- Note that pairs() doesn't maintain any sort of order in the
-- table, but Lua doesn't either so it doesn't matter.

local function formatValue(v)
  if tonumber(v) then
    return tonumber(v)
  elseif (v == "true" or v == "false") then
    return v == "true"
  else
    return v
  end
end

local function settingsFromStr(str)
  for k, v in str:gmatch("([^,]+)=([^,]+)") do
    settings[k] = formatValue(v)
  end
end

By the way - I watched some other examples with SetProjExtState that used "for k, v in pairs(s) do"
but if I want to save just ONE long string in SetProjExtState do I really need to do a for loop?

If you know it's one long string, then no - you should be able to plop it straight in there. Might need to replace any \ns with a placeholder and then swap them back on reload - I'm not sure.

Originally Posted by tompad

When closing project starting a new project my script window says open

Reaper doesn't send any events for the script to know that you've changed projects. AFAIK your best option would be to make use of the SWS Project Startup action or whatever it's called, but I don't think you'd be able to save into the previous project's ExtState that way.

[mpl]

Reaper doesn't send any events for the script to know that you've changed projects

in some cases this can be workarounded by tracking project name/path by EnumProjects() / GetProjectName()

[Lokasenna]

Originally Posted by mpl

in some cases this can be workarounded by tracking project name/path by EnumProjects() / GetProjectName()

I thought of that, but in this case you'd still be in the new project without having a chance to save your ExtState in the old one.

[spk77]

Originally Posted by mpl

in some cases this can be workarounded by tracking project name/path by EnumProjects() / GetProjectName()

Originally Posted by Lokasenna

I thought of that, but in this case you'd still be in the new project without having a chance to save your ExtState in the old one.

That is really annoying...here's an API request: https://forum.cockos.com/showthread....91#post1968991

Originally Posted by spk77

Two functions that would be nice to have:

1. reaper.atprojectexit(ReaProject proj, function)
   • Similar to reaper.atexit(function), but "function" would be run when the project is closing
2. reaper.SetProjExtState(string projectfilename, string extname, string key, string value)
   • SetProjExtState for offline project files

[tompad]

Originally Posted by spk77

That is really annoying...here's an API request: https://forum.cockos.com/showthread....91#post1968991

+1! Please, please please!

Where can I vote for this!?


Edit:
Or maybe I need to get back to saving to external file instead of waiting for a new API.
I guess Reaper devs have more importent things to do with the upcoming Reaper 6.
Its a shame though - I just managed to use Pickle and save to ProjExtState ;-(

[tompad]

Made a poll at Feature Request:

https://forum.cockos.com/showthread.php?t=211132

Lets go and vote!

[Meo-Ada Mespotine]

Originally Posted by mpl

in some cases this can be workarounded by tracking project name/path by EnumProjects() / GetProjectName()

Give me a few more weeks, I have some functions for tracking changed/added/deleted Projecttabs in my collection of helper-functions, that I'm going to release within the next few weeks.
Still finishing up the documentation for that...

The function in question will be:

boolean retval, integer countUnorderedProj, array unorderedProj, integer countNewProj, array newProj, integer countClosedProj, array closedProj = ultraschall.CheckForChangedProjectTabs(boolean update)

Returns if projecttabs have been changed due reordering, new projects or closed projects, since last calling this function.
Set update=true to update Ultraschall's internal project-monitoring-list or it will only return the changes since starting the API in this script or since the last time you used this function with parameter update set to true!

Returns false, -1 in case of error.

Returnvalues:
boolean retval - true, no changes in the projecttabs at all; false, either order, newprojects or closed project-changes
integer countUnorderedProj - the number of unordered projects
array unorderedProj - the unordered projects as ReaProjects
integer countNewProj - the number of new projects
array newProj - the new projects as ReaProjects
integer countClosedProj - the number of closed projects
array closedProj - the closed projects as ReaProjects

Parameters:
boolean update - true, update Ultraschall's internal projecttab-monitoring-list to the current state of all tabs
- false, don't update the internal projecttab-monitoring-list, so it will keep the "old" project-tab-state as checking-reference

[ThrashJazzAssassin]

There are a lot of references to Pickle in LBX Stripper

[tompad]

Originally Posted by ThrashJazzAssassin

There are a lot of references to Pickle in LBX Stripper

Thanks TrashJazzAssassin! I have fixed to use Pickle in my ToDoList-script,
however there where some issues when using using ProjExtState and changing
(closing) projects...Mespotine maybe have a solution though....I will check
it out when its ready - meanwhile GO AND VOTE! ;-)

Made a poll at Feature Request:

https://forum.cockos.com/showthread.php?t=211132

[tompad]

Originally Posted by mespotine

Give me a few more weeks, I have some functions for tracking changed/added/deleted Projecttabs in my collection of helper-functions, that I'm going to release within the next few weeks.

Sounds very interesting! Where is respotory? Github?

[Meo-Ada Mespotine]

It's not online yet, but I estimate I'll be ready mid-october. Just finishing docs and doing last bugfixes.
Will be worth the wait

[lb0]

Originally Posted by ThrashJazzAssassin

There are a lot of references to Pickle in LBX Stripper

Yep - I use it for some things still - but moved away from it for large amounts of data to my own dedicated routines (saving in files rather than the project) as this worked much much faster for my needs. For backwards compatibility I still use it for loading very old strips.

Regarding issues with changing projects (and using project tabs) - I solved this by having the script detect that the project has changed and saving the data before resetting the script for the current active project.

[tompad]

Originally Posted by lb0

Regarding issues with changing projects (and using project tabs) - I solved this by having the script detect that the project has changed and saving the data before resetting the script for the current active project.

Could you explain more? How can I detect that the project has changed?

[lb0]

Originally Posted by tompad

Could you explain more? How can I detect that the project has changed?

I created a unique (random) project id for the active project.

If one doesn't exist I create it and then save it in the project state. Also storing it in a variable.

Then every iteration of the defer loop - I would check the variable against the project state one. If they are different then I can assume it's a different project - and save the data before resetting for the different project (creating a new projectid if one doesn't exist, or reloading any necessary variables for an existing project from the state).

That's the gist anyway... I might have missed something vital as I cannot remember exactly how it worked when storing all the data in the project state, that got too slow for me - so I use external files instead.


When using external files to store data -

Things get more complex as you need to detect if the Save As or Save As New has been called - as the projectid will remain the same - then you also need to look at the project name also. This is not required if you're storing all data in the project state - but is required if you choose to use external data files.

[tompad]

Originally Posted by lb0

I created a unique (random) project id for the active project.

If one doesn't exist I create it and then save it in the project state. Also storing it in a variable.

Then every iteration of the defer loop - I would check the variable against the project state one. If they are different then I can assume it's a different project - and save the data before resetting for the different project (creating a new projectid if one doesn't exist, or reloading any necessary variables for an existing project from the state).

Thanks for telling this - interesting. The defer loop is something new for
me, have to take a look at it.

[Meo-Ada Mespotine]

Some basics:
Defer-loops are an alternative to while/for-loops. While and for loops, when running too long, "block" the GUI of Reaper at some point.
To overcome this, you can use defer-loops.
It will end the script, when reaching the defer-function, so Reaper can do the other tasks(other running scripts, updating GUI, etc) and return to your script, restarting it at the function you gave to the defer-function.
e.g. reaper.defer(main) will restart executing the script the next time from the function main().

The following code updates a counter and displays it into Reaper-Console-window, by running main().

Counter=0

function main()
   reaper.ClearConsole()
   reaper.ShowConsoleMsg(Counter)
   Counter=Counter+1

  -- now, we do the defering/looping
   reaper.defer(main)
end

main()

Some additional things:
defer-scripts are run about 30 times a second, so the execution of the main-function is never more often than 30 times. If you want to execute the code in the main function more often, you need to put it into a for-loop:

Counter=0

function main()
  for i=0, 10 do -- now we add to variable Counter 10 times more often per 
                 -- second,
                 -- means 300 times, as opposed to 30 times in the example 
                 -- before
    reaper.ClearConsole()
    reaper.ShowConsoleMsg(Counter)
    Counter=Counter+1

    -- now, we do the deferring/looping
    reaper.defer(main)
  end
end

main()

You can also use while-loops to never run the loop longer than a specific time. Use reaper.time_precise() to define a starting and endtime.


Another thing is, the first time the script is executed, it will run completely, until the end. From the second time onward, it will run only the deferred function.

Counter=0

function main()
  for i=0, 10 do -- now we add to variable Counter 10 times more often per 
                 -- second,
                 -- means 300 times, as opposed to 30 times in the example 
                 -- before
    reaper.ClearConsole()
    reaper.ShowConsoleMsg(Counter)
    Counter=Counter+1

    -- now, we do the deferring/looping
    reaper.defer(main)
  end
end

main()

reaper.MB("This Messagebox is shown only in the first run of the script!","",0)

Last thing is:
With every defer-cycle, the variables and returned values by functions will be updated. So with every defer-cycle, reaper.GetPlayPosition() will return the updated position of the playcursor.
These will be updated once, before going into the deferred-function. So the values stay the same, until they are updated the next defer-cycle.

function main()
   reaper.ClearConsole()
   reaper.ShowConsoleMsg(reaper.GetPlayPosition().."\n")
   for i=0, 10000 do end  
   reaper.ShowConsoleMsg(reaper.GetPlayPosition()) -- shows the same playposition as the ShowConsoleMsg() before, 
                                                   -- even if playposition has changed during the for-loop.
   
   -- now, we do the deferring/looping
   reaper.defer(main)
end

main()

PS:
When you want to run the defer-script only a limited number of times, use an if in the defer-line

Counter=0

function main()
   reaper.ClearConsole()
   reaper.ShowConsoleMsg(Counter)
   Counter=Counter+1

  -- now, we do the defering/looping
   if Counter<100 then reaper.defer(main) end -- will defer, until Counter is 100
end

main()

[tompad]

Originally Posted by mespotine

Some basics:
Defer-loops are an alternative to while/for-loops. While and for loops, when running too long, "block" the GUI of Reaper at some point.
To overcome this, you can use defer-loops.
It will end the script, when reaching the defer-function, so Reaper can do the other tasks(other running scripts, updating GUI, etc) and return to your script, restarting it at the function you gave to the defer-function.
e.g. reaper.defer(main) will restart executing the script the next time from the function main().

Vooooo! That was a lot to digest! But thanks anyway! :-)

[tompad]

@mespotine: Must I place the defer code in main or can I make
another function with defer?

Edit:
I have already a GUI.Main in my code - can I add another Main() then
or will it mess everything up??


I am thinking on using lb0's method to make my script
save to projextstate if project changes (new proj) and
then close my script window and reset the values in script.

Edit: ...or I wait a while for your helper functions :-)

[lexaproductions]

Originally Posted by mespotine

Some additional things:
defer-scripts are run about 30 times a second, so the execution of the main-function is never more often than 30 times. If you want to execute the code in the main function more often, you need to put it into a for-loop:

Counter=0

function main()
  for i=0, 10 do -- now we add to variable Counter 10 times more often per 
                 -- second,
                 -- means 300 times, as opposed to 30 times in the example 
                 -- before
    reaper.ClearConsole()
    reaper.ShowConsoleMsg(Counter)
    Counter=Counter+1

    -- now, we do the deferring/looping
    reaper.defer(main)
  end
end

main()

...

Last thing is:
With every defer-cycle, the variables and returned values by functions will be updated. So with every defer-cycle, reaper.GetPlayPosition() will return the updated position of the playcursor.
These will be updated once, before going into the deferred-function. So the values stay the same, until they are updated the next defer-cycle.

function main()
   reaper.ClearConsole()
   reaper.ShowConsoleMsg(reaper.GetPlayPosition().."\n")
   for i=0, 10000 do end  
   reaper.ShowConsoleMsg(reaper.GetPlayPosition()) -- shows the same playposition as the ShowConsoleMsg() before, 
                                                   -- even if playposition has changed during the for-loop.
   
   -- now, we do the deferring/looping
   reaper.defer(main)
end

main()

[/code]

Hey Mespotine,
Thanks for this explanation. I'm trying to make myself a very precise "On Next Measure" function, will this work? I'm not sure it does:

function on_NextMeasure(outfunction, last_msr)
	for i = 1, 1 do
		local isStopped = reaper.GetPlayState() == 0
		if isStopped then return end
		local _, msr = reaper.TimeMap2_timeToBeats(0, reaper.GetPlayPosition()+0.1) -- added 0.1 as timing adjust
		if last_msr and last_msr ~= msr then      -- if the measure number changed we're in another measure
			outfunction()
			return
		end
		last_msr = msr
	end
	reaper.defer(function() on_NextMeasure(outfunction, last_msr) end)
end

I need my "msr" variable to be updated and polled as fast as I can to get sub-milliseconds precision. But right now, it seems to be a hit and miss.

Thanks for your time.

[lexaproductions]

Bump

I hope Mespotine will see this...