Old 07-06-2017, 03:01 AM   #1
me2beats
Human being with feelings
 
me2beats's Avatar
 
Join Date: Jul 2015
Location: Yekaterinburg, Russia
Posts: 400
Default Get/SetTrackStateChunk() issues

Hey guys we (me and eugen2777) found some issues when using
GetTrackStateChunk() and SetTrackStateChunk()

Problems can be if you work with large track chunks (more than 4 mb)

At the moment we have the following solution (by eugen2777):

Code:
function GetTrackChunk(track)
  if not track then return end
  local fast_str, track_chunk
  fast_str = reaper.SNM_CreateFastString("")
  if reaper.SNM_GetSetObjectState(track, fast_str, false, false) then
    track_chunk = reaper.SNM_GetFastString(fast_str)
  end
  reaper.SNM_DeleteFastString(fast_str)  
  return track_chunk
end

function SetTrackChunk(track, track_chunk)
  if not (track and track_chunk) then return end
  local fast_str, ret
  fast_str = reaper.SNM_CreateFastString("")
  if reaper.SNM_SetFastString(fast_str, track_chunk) then
    ret = reaper.SNM_GetSetObjectState(track, fast_str, true, false)
  end
  reaper.SNM_DeleteFastString(fast_str)
  return ret
end
We recommend to use this code instead of GetTrackStateChunk() and SetTrackStateChunk() if possible.
__________________
My Reapack Repo
Donation

Last edited by me2beats; 07-06-2017 at 03:31 AM.
me2beats is offline   Reply With Quote
Old 07-06-2017, 03:54 AM   #2
snooks
Banned
 
Join Date: Sep 2015
Posts: 1,650
Default

Thanks, good work!
snooks is offline   Reply With Quote
Old 07-06-2017, 03:59 AM   #3
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 7,239
Default

What is the issue exactly? I remember to have issues in the past forcing me to work on chunks line by line in lua tables instead of strings. But at the end you have to pass a big string to set the chunk. How can o have a chunk so large to test it?.

Thank you and eugen2777 for the workaround.
heda is offline   Reply With Quote
Old 07-06-2017, 05:32 AM   #4
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 9,875
Default

Nice ! Thanks for sharing !

Can you or eugen put in on ReaTeam Templates repo ?
X-Raym is offline   Reply With Quote
Old 07-07-2017, 11:19 AM   #5
me2beats
Human being with feelings
 
me2beats's Avatar
 
Join Date: Jul 2015
Location: Yekaterinburg, Russia
Posts: 400
Default

Quote:
Originally Posted by heda View Post
What is the issue exactly?
For example track may lose several fx at the end of fx chain


Quote:
Originally Posted by X-Raym View Post
Can you or eugen put in on ReaTeam Templates repo ?
Hi X-Raym! I don't know to do this, can you put it to Templates repo yourself pls?
__________________
My Reapack Repo
Donation
me2beats is offline   Reply With Quote
Old 07-10-2017, 07:49 AM   #6
amagalma
Human being with feelings
 
amagalma's Avatar
 
Join Date: Apr 2011
Posts: 3,451
Default

Thank you for the functions!

Just to note that your workaround is quite a bit slower than the reaper function. For example, getting the chunk from 300 (almost empty) tracks:
- Reaper's function time: 0.013485364965163
- Eugen's function time: 0.18676601076731

Nonetheless, your workaround is invaluable when someone wants to be sure he doesn't encounter problems with tracks with large chunks and we thank you!
__________________
Most of my scripts can be found in ReaPack.
If you find them useful, a donation would be greatly appreciated! Thank you! :)
amagalma is offline   Reply With Quote
Old 07-10-2017, 03:40 PM   #7
James HE
Human being with feelings
 
James HE's Avatar
 
Join Date: Mar 2007
Location: I'm in a barn
Posts: 4,467
Default

Do we know if this is limited to .lua only? I remember testing some big chunks in eel and not having problems, might not have reached 4mb though.
James HE is offline   Reply With Quote
Old 07-20-2017, 01:28 AM   #8
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 4,171
Default

Just wanted to say thanks for this workaround.

This problem surfaced for one of my scripts and I couldn't initially work out why the data I was getting was truncated (yes to 4Mb).

The workaround worked beautifully! I think this is definitely something that needs fixing in the native API though.

Thanks again.
__________________
Projects - Reascripts - Lua:
Smart Knobs 2 | LBX Stripper | LBX Floating FX Positioner
Donate via Paypal | LBX Tools Website
lb0 is offline   Reply With Quote
Old 07-31-2017, 04:00 AM   #9
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 4,171
Default

Been playing around with these workarounds a bit now and have noticed some odd behaviour compared to working with track chunks using reapers getchunk functions.

If you add any new fx via the chunk - it will automatically generate a new FXID for any appended fx chunk. This is a little annoying as I was conditioning the inserted fx chunks with new ids before inserting them into the track chunk. Now I have to insert first then read the guid back to keep track of things. Anyway - it can be worked around.

The most odd behaviour is when I move an fx chunk to a different location in the chunk data - it will not reorder the FXID guids. Lets say I have four fx in the chunk with guids A,B,C,D.

If I reorder to place the D fx chunk at the beginning of the chain. I then check the chunk - the fx data has moved fine - but the FXID's still read in the same order - so now the chunk data D has had its GUID swapped with A. In fact all the guids have now shifted - so what should have gone:

A,B,C,D -> D,A,B,C

actually reads:

A,B,C,D -> A,B,C,D

but the guids are not attached to the same plugins as before (the plugins have moved in the fx list as required). Plugin D now has the guid originally assigned to plugin A, A now has B's GUID, B has C's and C has D's. :|

This makes it all very painful to keep track of.

Likewise, If I have plugins with GUIDs A,B,C,D

and I delete FX with guid A from the chunk.

I'm left with

A,B,C

So I need to write extra code to update all my control data with new guids.

Anyone know why it works like this? It seems this method is only good for changing fx offline status and things like that - but not reordering stuff (if you need to easily keep track).

When moving an fx up or down the fx list in the mixer - they retain their original guids as expected.
__________________
Projects - Reascripts - Lua:
Smart Knobs 2 | LBX Stripper | LBX Floating FX Positioner
Donate via Paypal | LBX Tools Website
lb0 is offline   Reply With Quote
Old 07-31-2017, 07:27 AM   #10
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 4,171
Default

Further to this - I've been looking at SNM_MoveOrRemoveTrackFX and this behaves in the same way.

Deleting the first fx in the chain results in all the guids changing for all remaining fx in the chain.

This is fine when doing simple stuff - but when I'm relying on GUIDs to keep track of fx plugins - and their GUIDs are likely to change using these functions - this is a major headache.

I can get around it internally in my script - but if an external script were to manipulate the guids (by using for example the above SNM call) - my script will have zero chance of retaining correct data!
__________________
Projects - Reascripts - Lua:
Smart Knobs 2 | LBX Stripper | LBX Floating FX Positioner
Donate via Paypal | LBX Tools Website
lb0 is offline   Reply With Quote
Old 07-31-2017, 08:54 AM   #11
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 7,239
Default

Quote:
Originally Posted by lb0 View Post
Further to this - I've been looking at SNM_MoveOrRemoveTrackFX and this behaves in the same way.
...
There is a problem indeed. SNM_MoveOrRemoveTrackFX should not be used until fixed:
https://forum.cockos.com/showthread.php?t=176623
heda is offline   Reply With Quote
Old 07-31-2017, 10:35 AM   #12
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 4,171
Default

Quote:
Originally Posted by heda View Post
There is a problem indeed. SNM_MoveOrRemoveTrackFX should not be used until fixed:
https://forum.cockos.com/showthread.php?t=176623
Aah - looks like I'm playing catch up with some of the quirks within reascript.

This one's really frustrating though. I noticed in the link you posted that you talked about writing your own routines to move items. Well I've done that and they work nicely with the reaper.GetTrackStateChunk routines - but using the workaround at the top of this thread to do the same thing is a headache for me due to GUIDs changing (and surely this is something you never want to happen).

Did you have any luck with this ?
__________________
Projects - Reascripts - Lua:
Smart Knobs 2 | LBX Stripper | LBX Floating FX Positioner
Donate via Paypal | LBX Tools Website
lb0 is offline   Reply With Quote
Old 07-31-2017, 11:35 AM   #13
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 7,239
Default

@lb0
I haven't tried the workaround in this first post because I have not encountered with chunks that large yet but it is on my list to test this. Save chunks functions shouldn't change anything in the string to save to the chunk. Strange!
heda is offline   Reply With Quote
Old 07-31-2017, 11:54 AM   #14
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 4,171
Default

Quote:
Originally Posted by heda View Post
@lb0
I haven't tried the workaround in this first post because I have not encountered with chunks that large yet but it is on my list to test this. Save chunks functions shouldn't change anything in the string to save to the chunk. Strange!
Ok thanks. Yeah - that's what I thought - but using the SNM functions seem to always preserve the order of GUIDs in the list - so reordering and deleting means plugin GUIDs will change - i'm literally taking an fx chunk - copying into another space in the chain and when setting the chunk back - the GUID's are rewritten as they were at the beginning but with the new fx data in place :/

I'm going to see if I can use the reaper.SetTrackStateChunk with the larger data - see if that suffers from the 4Mb limit.

I'm similar in that I hadn't come across chunks that big either - but a few people using my scripts have now reported losing plugins from the chain (or script crashing due to incomplete chunk data) which is definitely related to the truncated chunks.

I have put in a bug report for the main reaper.GetTrackStateChunk issue - see if it can be sorted out at source.
__________________
Projects - Reascripts - Lua:
Smart Knobs 2 | LBX Stripper | LBX Floating FX Positioner
Donate via Paypal | LBX Tools Website
lb0 is offline   Reply With Quote
Old 07-31-2017, 12:14 PM   #15
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 4,171
Default

Ok - at first - it appears the SetTrackStateChunk may not suffer the same limitations as the GetTrackStateChunk.

So I'm going to try out using the GetTrackChunk from the top of this thread (which works fine and brilliantly ) alongside the standard reaper.SetTrackStateChunk which seems to write back overly large chunks ok (in my quick limited tests).

If this is the case - then all should be good for my purposes

However - it should probably be mentioned (as I have) - that the SetTrackChunk function above using the SNM API calls can do strange things to GUIDs


EDIT - UPDATE:

Yep the native reaper.SetTrackStateChunk handles the large >4Mb chunk strings fine. (phew!)

So I would say go with the GetTrackChunk function provided at the top of this thread to replace reapers function - but stick with reaper's native settrackstatechunk function if you plan to do any reordering/removal of plugin data to avoid messing with the FXIDs.
__________________
Projects - Reascripts - Lua:
Smart Knobs 2 | LBX Stripper | LBX Floating FX Positioner
Donate via Paypal | LBX Tools Website

Last edited by lb0; 07-31-2017 at 02:29 PM.
lb0 is offline   Reply With Quote
Old 01-11-2018, 03:56 PM   #16
juliansader
Human being with feelings
 
Join Date: Jul 2009
Posts: 3,714
Default

The item version, GetItemStateChunk, has the same truncation bug.

(And the functions don't even return a "false" retval to indicate that a problem occurred.)
juliansader is offline   Reply With Quote
Old 01-12-2018, 04:21 PM   #17
tufb
Human being with feelings
 
Join Date: Dec 2017
Posts: 152
Default

Quote:
Originally Posted by heda View Post
What is the issue exactly? I remember to have issues in the past forcing me to work on chunks line by line in lua tables instead of strings. But at the end you have to pass a big string to set the chunk. How can o have a chunk so large to test it?.
If I understand the thread correctly, SetItemStateChunk may give no problems. Is the line by line table approach in Lua safe for large item Gets (I'd guess the answer is no as of the above post)?
tufb is offline   Reply With Quote
Old 01-13-2018, 04:14 PM   #18
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 7,239
Default

Quote:
Originally Posted by tufb View Post
If I understand the thread correctly, SetItemStateChunk may give no problems. Is the line by line table approach in Lua safe for large item Gets (I'd guess the answer is no as of the above post)?
I have no problems with it. I don't use any instruments that store large chunks. But I would like an answer by developers to this problem. Right now I get nightmares when I think about chunk manipulation.
heda is offline   Reply With Quote
Old 01-20-2018, 03:48 AM   #19
eugen2777
Human being with feelings
 
eugen2777's Avatar
 
Join Date: Aug 2012
Posts: 271
Default

The standard function GetChunk don't supports the of chunks more than 4M.
The standard function SetChunk supports the of chunks more than 4M.

As a result, I use a modified function to get chunk > 4m and a standard function to set chunk. It always works well.

Code:
--------------------------------------------------
function GetTrackChunk(track)
  if not track then return end
  -- Try standart function -----
  local ret, track_chunk = reaper.GetTrackStateChunk(track, "", false) -- isundo = false
  if ret and track_chunk and #track_chunk < 4194303 then return track_chunk end
  -- If chunk_size >= max_size, use wdl fast string --
  local fast_str = reaper.SNM_CreateFastString("")
  if reaper.SNM_GetSetObjectState(track, fast_str, false, false) then
    track_chunk = reaper.SNM_GetFastString(fast_str)
  end
  reaper.SNM_DeleteFastString(fast_str)
  return track_chunk
end

--------------------------------------------------
-- Set track chunk(allow > 4MB) ------------------
--------------------------------------------------
function SetTrackChunk(track, track_chunk)
  if not (track and track_chunk) then return end
  return reaper.SetTrackStateChunk(track, track_chunk, false)
end
The same can probably be used for item chunks.
__________________
ReaScripts
eugen2777 is offline   Reply With Quote
Old 01-22-2018, 02:33 AM   #20
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 7,239
Default

this one should be compatible with the reaper.GetTrackStateChunk. returning ret and track_chunk. For easy updating current code that uses reaper.GetTrackStateChunk

Code:
function GetTrackChunk(track)
  if not track then return end
  -- Try standard function -----
  local ret, track_chunk = reaper.GetTrackStateChunk(track, "", false) -- isundo = false
  if ret and track_chunk and #track_chunk < 4194303 then return ret, track_chunk end
  -- If chunk_size >= max_size, use wdl fast string --
  local fast_str = reaper.SNM_CreateFastString("")
  if reaper.SNM_GetSetObjectState(track, fast_str, false, false) then
    track_chunk = reaper.SNM_GetFastString(fast_str)
  end
  reaper.SNM_DeleteFastString(fast_str)
  if track_chunk then return true, track_chunk end
end

Last edited by heda; 01-22-2018 at 04:04 AM.
heda is offline   Reply With Quote
Old 02-23-2021, 06:45 AM   #21
nofish
Human being with feelings
 
nofish's Avatar
 
Join Date: Oct 2007
Location: home is where the heart is
Posts: 12,096
Default

Since it came up recently, this is apparently outdated info now.

Quote:
Originally Posted by whatsnew.txt
v5.95 - September 10 2018
+ ReaScript: remove size limitation and improve memory use for various APIs including Get*StateChunk(), GetSet*Info_String(), GetProjExtState(), GetSetProjectNotes(), MIDI_GetAllEvts()
nofish is offline   Reply With Quote
Old 02-23-2021, 05:10 PM   #22
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 7,239
Default

Quote:
Originally Posted by nofish View Post
Since it came up recently, this is apparently outdated info now.
hi nofish!
I think the problem of having so large strings is related to this other issue, even if the size limitation is solved, there are performance issues too: https://forum.cockos.com/showthread.php?t=248660
I think it would be great to have an API that could get the chunk without all the vst data that in most cases we don't need. Do you think it could be done in SWS? Since Justin doesn't know how to solve it apparently as you can read it there.
heda is offline   Reply With Quote
Old 02-23-2021, 05:19 PM   #23
Meo-Ada Mespotine
Human being with feelings
 
Meo-Ada Mespotine's Avatar
 
Join Date: May 2017
Location: Leipzig
Posts: 6,621
Default

Quote:
Originally Posted by heda View Post
Do you think it could be done in SWS?
Nope, as chunks must be created by Reaper itself. SWS can only use, what Reaper offers, which is limited to chunks with the fx-data included, unfortunately.

SWS could, however, get the statechunk and get rid of the fx-lines, which could be slightly faster in cpp, but setting it back would be a mess to code probably, as you would need to use your set-statechunk, take the fx-lines from the current statechunk and reinsert that back.
That means, a lot of memory needed for statechunks, who contain orchestra-libraries and such.
So our best hope are functions from the devs themselves, as they have full control over the data.
__________________
Use you/she/her.Ultraschall-Api Lua Api4Reaper - Donate, if you wish

On vacation for the time being...
Meo-Ada Mespotine is offline   Reply With Quote
Old 02-24-2021, 05:39 PM   #24
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 7,239
Default

Quote:
Originally Posted by Meo-Ada Mespotine View Post
Nope, as chunks must be created by Reaper itself. SWS can only use, what Reaper offers, which is limited to chunks with the fx-data included, unfortunately.

SWS could, however, get the statechunk and get rid of the fx-lines, which could be slightly faster in cpp, but setting it back would be a mess to code probably, as you would need to use your set-statechunk, take the fx-lines from the current statechunk and reinsert that back.
That means, a lot of memory needed for statechunks, who contain orchestra-libraries and such.
So our best hope are functions from the devs themselves, as they have full control over the data.
I think you are right
But the problem is not getting rid of the chunk data of the vst, I think the problem is in the Get the statechunk. SWS would also have to wait for the plugins to process the chunk before REAPER sends the data to SWS or to Lua... so it has to be fixed by REAPER.
heda is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 10:50 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.