Old 07-06-2017, 03:01 AM   #1
me2beats
Human being with feelings
 
me2beats's Avatar
 
Join Date: Jul 2015
Location: Yekaterinburg, Russia
Posts: 400
Default Get/SetTrackStateChunk() issues

Hey guys we (me and eugen2777) found some issues when using
GetTrackStateChunk() and SetTrackStateChunk()

Problems can be if you work with large track chunks (more than 4 mb)

At the moment we have the following solution (by eugen2777):

Code:
function GetTrackChunk(track)
  if not track then return end
  local fast_str, track_chunk
  fast_str = reaper.SNM_CreateFastString("")
  if reaper.SNM_GetSetObjectState(track, fast_str, false, false) then
    track_chunk = reaper.SNM_GetFastString(fast_str)
  end
  reaper.SNM_DeleteFastString(fast_str)  
  return track_chunk
end

function SetTrackChunk(track, track_chunk)
  if not (track and track_chunk) then return end
  local fast_str, ret
  fast_str = reaper.SNM_CreateFastString("")
  if reaper.SNM_SetFastString(fast_str, track_chunk) then
    ret = reaper.SNM_GetSetObjectState(track, fast_str, true, false)
  end
  reaper.SNM_DeleteFastString(fast_str)
  return ret
end
We recommend to use this code instead of GetTrackStateChunk() and SetTrackStateChunk() if possible.
__________________
My Reapack Repo
Donation

Last edited by me2beats; 07-06-2017 at 03:31 AM.
me2beats is offline   Reply With Quote
Old 07-06-2017, 03:54 AM   #2
snooks
Human being with feelings
 
Join Date: Sep 2015
Posts: 1,636
Default

Thanks, good work!
snooks is offline   Reply With Quote
Old 07-06-2017, 03:59 AM   #3
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 5,433
Default

What is the issue exactly? I remember to have issues in the past forcing me to work on chunks line by line in lua tables instead of strings. But at the end you have to pass a big string to set the chunk. How can o have a chunk so large to test it?.

Thank you and eugen2777 for the workaround.
heda is offline   Reply With Quote
Old 07-06-2017, 05:32 AM   #4
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 6,041
Default

Nice ! Thanks for sharing !

Can you or eugen put in on ReaTeam Templates repo ?
X-Raym is offline   Reply With Quote
Old 07-07-2017, 11:19 AM   #5
me2beats
Human being with feelings
 
me2beats's Avatar
 
Join Date: Jul 2015
Location: Yekaterinburg, Russia
Posts: 400
Default

Quote:
Originally Posted by heda View Post
What is the issue exactly?
For example track may lose several fx at the end of fx chain


Quote:
Originally Posted by X-Raym View Post
Can you or eugen put in on ReaTeam Templates repo ?
Hi X-Raym! I don't know to do this, can you put it to Templates repo yourself pls?
__________________
My Reapack Repo
Donation
me2beats is offline   Reply With Quote
Old 07-10-2017, 07:49 AM   #6
amagalma
Human being with feelings
 
Join Date: Apr 2011
Posts: 1,503
Default

Thank you for the functions!

Just to note that your workaround is quite a bit slower than the reaper function. For example, getting the chunk from 300 (almost empty) tracks:
- Reaper's function time: 0.013485364965163
- Eugen's function time: 0.18676601076731

Nonetheless, your workaround is invaluable when someone wants to be sure he doesn't encounter problems with tracks with large chunks and we thank you!
amagalma is offline   Reply With Quote
Old 07-10-2017, 03:40 PM   #7
James HE
Human being with feelings
 
James HE's Avatar
 
Join Date: Mar 2007
Location: I'm in a barn
Posts: 4,415
Default

Do we know if this is limited to .lua only? I remember testing some big chunks in eel and not having problems, might not have reached 4mb though.
James HE is offline   Reply With Quote
Old 07-20-2017, 01:28 AM   #8
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 3,111
Default

Just wanted to say thanks for this workaround.

This problem surfaced for one of my scripts and I couldn't initially work out why the data I was getting was truncated (yes to 4Mb).

The workaround worked beautifully! I think this is definitely something that needs fixing in the native API though.

Thanks again.
__________________
Projects - Reascripts - Lua:
LBX Stripper | LBX Chaos Engine | LBX Floating FX Positioner | LBX SRD Smart Knobs
Donate via Paypal
lb0 is offline   Reply With Quote
Old 07-31-2017, 04:00 AM   #9
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 3,111
Default

Been playing around with these workarounds a bit now and have noticed some odd behaviour compared to working with track chunks using reapers getchunk functions.

If you add any new fx via the chunk - it will automatically generate a new FXID for any appended fx chunk. This is a little annoying as I was conditioning the inserted fx chunks with new ids before inserting them into the track chunk. Now I have to insert first then read the guid back to keep track of things. Anyway - it can be worked around.

The most odd behaviour is when I move an fx chunk to a different location in the chunk data - it will not reorder the FXID guids. Lets say I have four fx in the chunk with guids A,B,C,D.

If I reorder to place the D fx chunk at the beginning of the chain. I then check the chunk - the fx data has moved fine - but the FXID's still read in the same order - so now the chunk data D has had its GUID swapped with A. In fact all the guids have now shifted - so what should have gone:

A,B,C,D -> D,A,B,C

actually reads:

A,B,C,D -> A,B,C,D

but the guids are not attached to the same plugins as before (the plugins have moved in the fx list as required). Plugin D now has the guid originally assigned to plugin A, A now has B's GUID, B has C's and C has D's. :|

This makes it all very painful to keep track of.

Likewise, If I have plugins with GUIDs A,B,C,D

and I delete FX with guid A from the chunk.

I'm left with

A,B,C

So I need to write extra code to update all my control data with new guids.

Anyone know why it works like this? It seems this method is only good for changing fx offline status and things like that - but not reordering stuff (if you need to easily keep track).

When moving an fx up or down the fx list in the mixer - they retain their original guids as expected.
__________________
Projects - Reascripts - Lua:
LBX Stripper | LBX Chaos Engine | LBX Floating FX Positioner | LBX SRD Smart Knobs
Donate via Paypal
lb0 is offline   Reply With Quote
Old 07-31-2017, 07:27 AM   #10
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 3,111
Default

Further to this - I've been looking at SNM_MoveOrRemoveTrackFX and this behaves in the same way.

Deleting the first fx in the chain results in all the guids changing for all remaining fx in the chain.

This is fine when doing simple stuff - but when I'm relying on GUIDs to keep track of fx plugins - and their GUIDs are likely to change using these functions - this is a major headache.

I can get around it internally in my script - but if an external script were to manipulate the guids (by using for example the above SNM call) - my script will have zero chance of retaining correct data!
__________________
Projects - Reascripts - Lua:
LBX Stripper | LBX Chaos Engine | LBX Floating FX Positioner | LBX SRD Smart Knobs
Donate via Paypal
lb0 is offline   Reply With Quote
Old 07-31-2017, 08:54 AM   #11
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 5,433
Default

Quote:
Originally Posted by lb0 View Post
Further to this - I've been looking at SNM_MoveOrRemoveTrackFX and this behaves in the same way.
...
There is a problem indeed. SNM_MoveOrRemoveTrackFX should not be used until fixed:
https://forum.cockos.com/showthread.php?t=176623
heda is offline   Reply With Quote
Old 07-31-2017, 10:35 AM   #12
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 3,111
Default

Quote:
Originally Posted by heda View Post
There is a problem indeed. SNM_MoveOrRemoveTrackFX should not be used until fixed:
https://forum.cockos.com/showthread.php?t=176623
Aah - looks like I'm playing catch up with some of the quirks within reascript.

This one's really frustrating though. I noticed in the link you posted that you talked about writing your own routines to move items. Well I've done that and they work nicely with the reaper.GetTrackStateChunk routines - but using the workaround at the top of this thread to do the same thing is a headache for me due to GUIDs changing (and surely this is something you never want to happen).

Did you have any luck with this ?
__________________
Projects - Reascripts - Lua:
LBX Stripper | LBX Chaos Engine | LBX Floating FX Positioner | LBX SRD Smart Knobs
Donate via Paypal
lb0 is offline   Reply With Quote
Old 07-31-2017, 11:35 AM   #13
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 5,433
Default

@lb0
I haven't tried the workaround in this first post because I have not encountered with chunks that large yet but it is on my list to test this. Save chunks functions shouldn't change anything in the string to save to the chunk. Strange!
heda is offline   Reply With Quote
Old 07-31-2017, 11:54 AM   #14
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 3,111
Default

Quote:
Originally Posted by heda View Post
@lb0
I haven't tried the workaround in this first post because I have not encountered with chunks that large yet but it is on my list to test this. Save chunks functions shouldn't change anything in the string to save to the chunk. Strange!
Ok thanks. Yeah - that's what I thought - but using the SNM functions seem to always preserve the order of GUIDs in the list - so reordering and deleting means plugin GUIDs will change - i'm literally taking an fx chunk - copying into another space in the chain and when setting the chunk back - the GUID's are rewritten as they were at the beginning but with the new fx data in place :/

I'm going to see if I can use the reaper.SetTrackStateChunk with the larger data - see if that suffers from the 4Mb limit.

I'm similar in that I hadn't come across chunks that big either - but a few people using my scripts have now reported losing plugins from the chain (or script crashing due to incomplete chunk data) which is definitely related to the truncated chunks.

I have put in a bug report for the main reaper.GetTrackStateChunk issue - see if it can be sorted out at source.
__________________
Projects - Reascripts - Lua:
LBX Stripper | LBX Chaos Engine | LBX Floating FX Positioner | LBX SRD Smart Knobs
Donate via Paypal
lb0 is offline   Reply With Quote
Old 07-31-2017, 12:14 PM   #15
lb0
Human being with feelings
 
Join Date: Apr 2014
Posts: 3,111
Default

Ok - at first - it appears the SetTrackStateChunk may not suffer the same limitations as the GetTrackStateChunk.

So I'm going to try out using the GetTrackChunk from the top of this thread (which works fine and brilliantly ) alongside the standard reaper.SetTrackStateChunk which seems to write back overly large chunks ok (in my quick limited tests).

If this is the case - then all should be good for my purposes

However - it should probably be mentioned (as I have) - that the SetTrackChunk function above using the SNM API calls can do strange things to GUIDs


EDIT - UPDATE:

Yep the native reaper.SetTrackStateChunk handles the large >4Mb chunk strings fine. (phew!)

So I would say go with the GetTrackChunk function provided at the top of this thread to replace reapers function - but stick with reaper's native settrackstatechunk function if you plan to do any reordering/removal of plugin data to avoid messing with the FXIDs.
__________________
Projects - Reascripts - Lua:
LBX Stripper | LBX Chaos Engine | LBX Floating FX Positioner | LBX SRD Smart Knobs
Donate via Paypal

Last edited by lb0; 07-31-2017 at 02:29 PM.
lb0 is offline   Reply With Quote
Old 01-11-2018, 03:56 PM   #16
juliansader
Human being with feelings
 
Join Date: Jul 2009
Posts: 2,757
Default

The item version, GetItemStateChunk, has the same truncation bug.

(And the functions don't even return a "false" retval to indicate that a problem occurred.)
juliansader is offline   Reply With Quote
Old 01-12-2018, 04:21 PM   #17
tufb
Human being with feelings
 
Join Date: Dec 2017
Location: Germany
Posts: 49
Default

Quote:
Originally Posted by heda View Post
What is the issue exactly? I remember to have issues in the past forcing me to work on chunks line by line in lua tables instead of strings. But at the end you have to pass a big string to set the chunk. How can o have a chunk so large to test it?.
If I understand the thread correctly, SetItemStateChunk may give no problems. Is the line by line table approach in Lua safe for large item Gets (I'd guess the answer is no as of the above post)?
tufb is offline   Reply With Quote
Old 01-13-2018, 04:14 PM   #18
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 5,433
Default

Quote:
Originally Posted by tufb View Post
If I understand the thread correctly, SetItemStateChunk may give no problems. Is the line by line table approach in Lua safe for large item Gets (I'd guess the answer is no as of the above post)?
I have no problems with it. I don't use any instruments that store large chunks. But I would like an answer by developers to this problem. Right now I get nightmares when I think about chunk manipulation.
heda is offline   Reply With Quote
Old 01-20-2018, 03:48 AM   #19
eugen2777
Human being with feelings
 
eugen2777's Avatar
 
Join Date: Aug 2012
Posts: 271
Default

The standard function GetChunk don't supports the of chunks more than 4M.
The standard function SetChunk supports the of chunks more than 4M.

As a result, I use a modified function to get chunk > 4m and a standard function to set chunk. It always works well.

Code:
--------------------------------------------------
function GetTrackChunk(track)
  if not track then return end
  -- Try standart function -----
  local ret, track_chunk = reaper.GetTrackStateChunk(track, "", false) -- isundo = false
  if ret and track_chunk and #track_chunk < 4194303 then return track_chunk end
  -- If chunk_size >= max_size, use wdl fast string --
  local fast_str = reaper.SNM_CreateFastString("")
  if reaper.SNM_GetSetObjectState(track, fast_str, false, false) then
    track_chunk = reaper.SNM_GetFastString(fast_str)
  end
  reaper.SNM_DeleteFastString(fast_str)
  return track_chunk
end

--------------------------------------------------
-- Set track chunk(allow > 4MB) ------------------
--------------------------------------------------
function SetTrackChunk(track, track_chunk)
  if not (track and track_chunk) then return end
  return reaper.SetTrackStateChunk(track, track_chunk, false)
end
The same can probably be used for item chunks.
__________________
ReaScripts
eugen2777 is offline   Reply With Quote
Old 01-22-2018, 02:33 AM   #20
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 5,433
Default

this one should be compatible with the reaper.GetTrackStateChunk. returning ret and track_chunk. For easy updating current code that uses reaper.GetTrackStateChunk

Code:
function GetTrackChunk(track)
  if not track then return end
  -- Try standard function -----
  local ret, track_chunk = reaper.GetTrackStateChunk(track, "", false) -- isundo = false
  if ret and track_chunk and #track_chunk < 4194303 then return ret, track_chunk end
  -- If chunk_size >= max_size, use wdl fast string --
  local fast_str = reaper.SNM_CreateFastString("")
  if reaper.SNM_GetSetObjectState(track, fast_str, false, false) then
    track_chunk = reaper.SNM_GetFastString(fast_str)
  end
  reaper.SNM_DeleteFastString(fast_str)
  if track_chunk then return true, track_chunk end
end

Last edited by heda; 01-22-2018 at 04:04 AM.
heda is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 08:59 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.