|
|
|
07-06-2017, 03:01 AM
|
#1
|
Human being with feelings
Join Date: Jul 2015
Location: Yekaterinburg, Russia
Posts: 400
|
Get/SetTrackStateChunk() issues
Hey guys we (me and eugen2777) found some issues when using
GetTrackStateChunk() and SetTrackStateChunk()
Problems can be if you work with large track chunks (more than 4 mb)
At the moment we have the following solution (by eugen2777):
Code:
function GetTrackChunk(track)
if not track then return end
local fast_str, track_chunk
fast_str = reaper.SNM_CreateFastString("")
if reaper.SNM_GetSetObjectState(track, fast_str, false, false) then
track_chunk = reaper.SNM_GetFastString(fast_str)
end
reaper.SNM_DeleteFastString(fast_str)
return track_chunk
end
function SetTrackChunk(track, track_chunk)
if not (track and track_chunk) then return end
local fast_str, ret
fast_str = reaper.SNM_CreateFastString("")
if reaper.SNM_SetFastString(fast_str, track_chunk) then
ret = reaper.SNM_GetSetObjectState(track, fast_str, true, false)
end
reaper.SNM_DeleteFastString(fast_str)
return ret
end
We recommend to use this code instead of GetTrackStateChunk() and SetTrackStateChunk() if possible.
Last edited by me2beats; 07-06-2017 at 03:31 AM.
|
|
|
07-06-2017, 03:54 AM
|
#2
|
Banned
Join Date: Sep 2015
Posts: 1,650
|
Thanks, good work!
|
|
|
07-06-2017, 03:59 AM
|
#3
|
Human being with feelings
Join Date: Jun 2012
Location: Spain
Posts: 7,268
|
What is the issue exactly? I remember to have issues in the past forcing me to work on chunks line by line in lua tables instead of strings. But at the end you have to pass a big string to set the chunk. How can o have a chunk so large to test it?.
Thank you and eugen2777 for the workaround.
|
|
|
07-06-2017, 05:32 AM
|
#4
|
Human being with feelings
Join Date: Apr 2013
Location: France
Posts: 9,900
|
Nice ! Thanks for sharing !
Can you or eugen put in on ReaTeam Templates repo ?
|
|
|
07-07-2017, 11:19 AM
|
#5
|
Human being with feelings
Join Date: Jul 2015
Location: Yekaterinburg, Russia
Posts: 400
|
Quote:
Originally Posted by heda
What is the issue exactly?
|
For example track may lose several fx at the end of fx chain
Quote:
Originally Posted by X-Raym
|
Hi X-Raym! I don't know to do this, can you put it to Templates repo yourself pls?
|
|
|
07-10-2017, 07:49 AM
|
#6
|
Human being with feelings
Join Date: Apr 2011
Posts: 3,458
|
Thank you for the functions!
Just to note that your workaround is quite a bit slower than the reaper function. For example, getting the chunk from 300 (almost empty) tracks:
- Reaper's function time: 0.013485364965163
- Eugen's function time: 0.18676601076731
Nonetheless, your workaround is invaluable when someone wants to be sure he doesn't encounter problems with tracks with large chunks and we thank you!
|
|
|
07-10-2017, 03:40 PM
|
#7
|
Human being with feelings
Join Date: Mar 2007
Location: I'm in a barn
Posts: 4,467
|
Do we know if this is limited to .lua only? I remember testing some big chunks in eel and not having problems, might not have reached 4mb though.
|
|
|
07-20-2017, 01:28 AM
|
#8
|
Human being with feelings
Join Date: Apr 2014
Posts: 4,175
|
Just wanted to say thanks for this workaround.
This problem surfaced for one of my scripts and I couldn't initially work out why the data I was getting was truncated (yes to 4Mb).
The workaround worked beautifully! I think this is definitely something that needs fixing in the native API though.
Thanks again.
|
|
|
07-31-2017, 04:00 AM
|
#9
|
Human being with feelings
Join Date: Apr 2014
Posts: 4,175
|
Been playing around with these workarounds a bit now and have noticed some odd behaviour compared to working with track chunks using reapers getchunk functions.
If you add any new fx via the chunk - it will automatically generate a new FXID for any appended fx chunk. This is a little annoying as I was conditioning the inserted fx chunks with new ids before inserting them into the track chunk. Now I have to insert first then read the guid back to keep track of things. Anyway - it can be worked around.
The most odd behaviour is when I move an fx chunk to a different location in the chunk data - it will not reorder the FXID guids. Lets say I have four fx in the chunk with guids A,B,C,D.
If I reorder to place the D fx chunk at the beginning of the chain. I then check the chunk - the fx data has moved fine - but the FXID's still read in the same order - so now the chunk data D has had its GUID swapped with A. In fact all the guids have now shifted - so what should have gone:
A,B,C,D -> D,A,B,C
actually reads:
A,B,C,D -> A,B,C,D
but the guids are not attached to the same plugins as before (the plugins have moved in the fx list as required). Plugin D now has the guid originally assigned to plugin A, A now has B's GUID, B has C's and C has D's. :|
This makes it all very painful to keep track of.
Likewise, If I have plugins with GUIDs A,B,C,D
and I delete FX with guid A from the chunk.
I'm left with
A,B,C
So I need to write extra code to update all my control data with new guids.
Anyone know why it works like this? It seems this method is only good for changing fx offline status and things like that - but not reordering stuff (if you need to easily keep track).
When moving an fx up or down the fx list in the mixer - they retain their original guids as expected.
|
|
|
07-31-2017, 07:27 AM
|
#10
|
Human being with feelings
Join Date: Apr 2014
Posts: 4,175
|
Further to this - I've been looking at SNM_MoveOrRemoveTrackFX and this behaves in the same way.
Deleting the first fx in the chain results in all the guids changing for all remaining fx in the chain.
This is fine when doing simple stuff - but when I'm relying on GUIDs to keep track of fx plugins - and their GUIDs are likely to change using these functions - this is a major headache.
I can get around it internally in my script - but if an external script were to manipulate the guids (by using for example the above SNM call) - my script will have zero chance of retaining correct data!
|
|
|
07-31-2017, 08:54 AM
|
#11
|
Human being with feelings
Join Date: Jun 2012
Location: Spain
Posts: 7,268
|
Quote:
Originally Posted by lb0
Further to this - I've been looking at SNM_MoveOrRemoveTrackFX and this behaves in the same way.
...
|
There is a problem indeed. SNM_MoveOrRemoveTrackFX should not be used until fixed:
https://forum.cockos.com/showthread.php?t=176623
|
|
|
07-31-2017, 10:35 AM
|
#12
|
Human being with feelings
Join Date: Apr 2014
Posts: 4,175
|
Quote:
Originally Posted by heda
|
Aah - looks like I'm playing catch up with some of the quirks within reascript.
This one's really frustrating though. I noticed in the link you posted that you talked about writing your own routines to move items. Well I've done that and they work nicely with the reaper.GetTrackStateChunk routines - but using the workaround at the top of this thread to do the same thing is a headache for me due to GUIDs changing (and surely this is something you never want to happen).
Did you have any luck with this ?
|
|
|
07-31-2017, 11:35 AM
|
#13
|
Human being with feelings
Join Date: Jun 2012
Location: Spain
Posts: 7,268
|
@lb0
I haven't tried the workaround in this first post because I have not encountered with chunks that large yet but it is on my list to test this. Save chunks functions shouldn't change anything in the string to save to the chunk. Strange!
|
|
|
07-31-2017, 11:54 AM
|
#14
|
Human being with feelings
Join Date: Apr 2014
Posts: 4,175
|
Quote:
Originally Posted by heda
@lb0
I haven't tried the workaround in this first post because I have not encountered with chunks that large yet but it is on my list to test this. Save chunks functions shouldn't change anything in the string to save to the chunk. Strange!
|
Ok thanks. Yeah - that's what I thought - but using the SNM functions seem to always preserve the order of GUIDs in the list - so reordering and deleting means plugin GUIDs will change - i'm literally taking an fx chunk - copying into another space in the chain and when setting the chunk back - the GUID's are rewritten as they were at the beginning but with the new fx data in place :/
I'm going to see if I can use the reaper.SetTrackStateChunk with the larger data - see if that suffers from the 4Mb limit.
I'm similar in that I hadn't come across chunks that big either - but a few people using my scripts have now reported losing plugins from the chain (or script crashing due to incomplete chunk data) which is definitely related to the truncated chunks.
I have put in a bug report for the main reaper.GetTrackStateChunk issue - see if it can be sorted out at source.
|
|
|
07-31-2017, 12:14 PM
|
#15
|
Human being with feelings
Join Date: Apr 2014
Posts: 4,175
|
Ok - at first - it appears the SetTrackStateChunk may not suffer the same limitations as the GetTrackStateChunk.
So I'm going to try out using the GetTrackChunk from the top of this thread (which works fine and brilliantly ) alongside the standard reaper.SetTrackStateChunk which seems to write back overly large chunks ok (in my quick limited tests).
If this is the case - then all should be good for my purposes
However - it should probably be mentioned (as I have) - that the SetTrackChunk function above using the SNM API calls can do strange things to GUIDs
EDIT - UPDATE:
Yep the native reaper.SetTrackStateChunk handles the large >4Mb chunk strings fine. (phew!)
So I would say go with the GetTrackChunk function provided at the top of this thread to replace reapers function - but stick with reaper's native settrackstatechunk function if you plan to do any reordering/removal of plugin data to avoid messing with the FXIDs.
Last edited by lb0; 07-31-2017 at 02:29 PM.
|
|
|
01-11-2018, 03:56 PM
|
#16
|
Human being with feelings
Join Date: Jul 2009
Posts: 3,714
|
The item version, GetItemStateChunk, has the same truncation bug.
(And the functions don't even return a "false" retval to indicate that a problem occurred.)
|
|
|
01-12-2018, 04:21 PM
|
#17
|
Human being with feelings
Join Date: Dec 2017
Posts: 152
|
Quote:
Originally Posted by heda
What is the issue exactly? I remember to have issues in the past forcing me to work on chunks line by line in lua tables instead of strings. But at the end you have to pass a big string to set the chunk. How can o have a chunk so large to test it?.
|
If I understand the thread correctly, SetItemStateChunk may give no problems. Is the line by line table approach in Lua safe for large item Gets (I'd guess the answer is no as of the above post)?
|
|
|
01-13-2018, 04:14 PM
|
#18
|
Human being with feelings
Join Date: Jun 2012
Location: Spain
Posts: 7,268
|
Quote:
Originally Posted by tufb
If I understand the thread correctly, SetItemStateChunk may give no problems. Is the line by line table approach in Lua safe for large item Gets (I'd guess the answer is no as of the above post)?
|
I have no problems with it. I don't use any instruments that store large chunks. But I would like an answer by developers to this problem. Right now I get nightmares when I think about chunk manipulation.
|
|
|
01-20-2018, 03:48 AM
|
#19
|
Human being with feelings
Join Date: Aug 2012
Posts: 271
|
The standard function GetChunk don't supports the of chunks more than 4M.
The standard function SetChunk supports the of chunks more than 4M.
As a result, I use a modified function to get chunk > 4m and a standard function to set chunk. It always works well.
Code:
--------------------------------------------------
function GetTrackChunk(track)
if not track then return end
-- Try standart function -----
local ret, track_chunk = reaper.GetTrackStateChunk(track, "", false) -- isundo = false
if ret and track_chunk and #track_chunk < 4194303 then return track_chunk end
-- If chunk_size >= max_size, use wdl fast string --
local fast_str = reaper.SNM_CreateFastString("")
if reaper.SNM_GetSetObjectState(track, fast_str, false, false) then
track_chunk = reaper.SNM_GetFastString(fast_str)
end
reaper.SNM_DeleteFastString(fast_str)
return track_chunk
end
--------------------------------------------------
-- Set track chunk(allow > 4MB) ------------------
--------------------------------------------------
function SetTrackChunk(track, track_chunk)
if not (track and track_chunk) then return end
return reaper.SetTrackStateChunk(track, track_chunk, false)
end
The same can probably be used for item chunks.
|
|
|
01-22-2018, 02:33 AM
|
#20
|
Human being with feelings
Join Date: Jun 2012
Location: Spain
Posts: 7,268
|
this one should be compatible with the reaper.GetTrackStateChunk. returning ret and track_chunk. For easy updating current code that uses reaper.GetTrackStateChunk
Code:
function GetTrackChunk(track)
if not track then return end
-- Try standard function -----
local ret, track_chunk = reaper.GetTrackStateChunk(track, "", false) -- isundo = false
if ret and track_chunk and #track_chunk < 4194303 then return ret, track_chunk end
-- If chunk_size >= max_size, use wdl fast string --
local fast_str = reaper.SNM_CreateFastString("")
if reaper.SNM_GetSetObjectState(track, fast_str, false, false) then
track_chunk = reaper.SNM_GetFastString(fast_str)
end
reaper.SNM_DeleteFastString(fast_str)
if track_chunk then return true, track_chunk end
end
Last edited by heda; 01-22-2018 at 04:04 AM.
|
|
|
02-23-2021, 06:45 AM
|
#21
|
Human being with feelings
Join Date: Oct 2007
Location: home is where the heart is
Posts: 12,110
|
Since it came up recently, this is apparently outdated info now.
Quote:
Originally Posted by whatsnew.txt
v5.95 - September 10 2018
+ ReaScript: remove size limitation and improve memory use for various APIs including Get*StateChunk(), GetSet*Info_String(), GetProjExtState(), GetSetProjectNotes(), MIDI_GetAllEvts()
|
|
|
|
02-23-2021, 05:10 PM
|
#22
|
Human being with feelings
Join Date: Jun 2012
Location: Spain
Posts: 7,268
|
Quote:
Originally Posted by nofish
Since it came up recently, this is apparently outdated info now.
|
hi nofish!
I think the problem of having so large strings is related to this other issue, even if the size limitation is solved, there are performance issues too: https://forum.cockos.com/showthread.php?t=248660
I think it would be great to have an API that could get the chunk without all the vst data that in most cases we don't need. Do you think it could be done in SWS? Since Justin doesn't know how to solve it apparently as you can read it there.
|
|
|
02-23-2021, 05:19 PM
|
#23
|
Human being with feelings
Join Date: May 2017
Location: Leipzig
Posts: 6,630
|
Quote:
Originally Posted by heda
Do you think it could be done in SWS?
|
Nope, as chunks must be created by Reaper itself. SWS can only use, what Reaper offers, which is limited to chunks with the fx-data included, unfortunately.
SWS could, however, get the statechunk and get rid of the fx-lines, which could be slightly faster in cpp, but setting it back would be a mess to code probably, as you would need to use your set-statechunk, take the fx-lines from the current statechunk and reinsert that back.
That means, a lot of memory needed for statechunks, who contain orchestra-libraries and such.
So our best hope are functions from the devs themselves, as they have full control over the data.
|
|
|
02-24-2021, 05:39 PM
|
#24
|
Human being with feelings
Join Date: Jun 2012
Location: Spain
Posts: 7,268
|
Quote:
Originally Posted by Meo-Ada Mespotine
Nope, as chunks must be created by Reaper itself. SWS can only use, what Reaper offers, which is limited to chunks with the fx-data included, unfortunately.
SWS could, however, get the statechunk and get rid of the fx-lines, which could be slightly faster in cpp, but setting it back would be a mess to code probably, as you would need to use your set-statechunk, take the fx-lines from the current statechunk and reinsert that back.
That means, a lot of memory needed for statechunks, who contain orchestra-libraries and such.
So our best hope are functions from the devs themselves, as they have full control over the data.
|
I think you are right
But the problem is not getting rid of the chunk data of the vst, I think the problem is in the Get the statechunk. SWS would also have to wait for the plugins to process the chunk before REAPER sends the data to SWS or to Lua... so it has to be fixed by REAPER.
|
|
|
Thread Tools |
|
Display Modes |
Linear Mode
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
All times are GMT -7. The time now is 05:23 AM.
|