Old 10-24-2016, 06:00 AM   #1
tack
Human being with feelings
 
tack's Avatar
 
Join Date: Jan 2014
Location: Ontario, Canada
Posts: 1,619
Default How to store per-track user data?

Hi Reaper gurus,

I'm not able to find any APIs that allow one to associate user data to a track. If I've missed something, please administer a proper cluebatting.

Meanwhile, there are of course options to store either global state or project-level state, and one could store project state using the track UUID as the key.

There are two problems with this approach though:
1. User data is not carried over when the track is duplicated
2. The script must periodically sweep the stored data to make sure that data is properly cleaned up for deleted tracks.

#2 is a minor inconvenience, but #1 is something I don't have a solution for.

Are there any APIs to deal with this use-case, or maybe some common patterns that others use to solve the problem?

Thanks!
tack is online now   Reply With Quote
Old 10-24-2016, 07:51 AM   #2
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 9,875
Default

You can't add custom data attributes to a track, you have to deal with GUID and ProjectExtState

You can make custom actions to copy paste your particular data from one track to another, or most accurately, to associate a new track GUI to a certain ProjExtState data.

What are you trying to do ?
X-Raym is offline   Reply With Quote
Old 10-24-2016, 12:24 PM   #3
tack
Human being with feelings
 
tack's Avatar
 
Join Date: Jan 2014
Location: Ontario, Canada
Posts: 1,619
Default

I have a long-running script that provides a UI related to the current track. The user is allowed to make changes in the UI that are per-track.

I wanted it so that if the user duplicates the track those changes are carried along. I could add a custom action to duplicate tracks to preserve the metadata associated with my script, but that does require users to switch their shortcuts to it, and of course only works for exactly one such script.

Thanks X-Raym!
tack is online now   Reply With Quote
Old 10-24-2016, 12:26 PM   #4
Xenakios
Human being with feelings
 
Xenakios's Avatar
 
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 8,062
Default

It would sure be nice to be able to add custom properties into the various objects in Reaper : tracks, items, takes, fx instances...
__________________
I am no longer part of the REAPER community. Please don't contact me with any REAPER-related issues.
Xenakios is offline   Reply With Quote
Old 10-24-2016, 05:29 PM   #5
tack
Human being with feelings
 
tack's Avatar
 
Join Date: Jan 2014
Location: Ontario, Canada
Posts: 1,619
Default

Yeah. It's not the first time I've wanted to do this.

Meanwhile, to handle the single track duplication use-case, I guess I can leverage the fact that duplicating the tracks automatically selects the newly created track. I can detect the track change, and if the new track GUID isn't found in the project state and it has the same name as the previously select tracked, then my script can clone the userdata to the new track GUID.

I think that's a workable pattern.
tack is online now   Reply With Quote
Old 03-24-2018, 05:37 AM   #6
spk77
Human being with feelings
 
Join Date: Aug 2012
Location: Finland
Posts: 2,668
Default

Quote:
Originally Posted by Xenakios View Post
It would sure be nice to be able to add custom properties into the various objects in Reaper : tracks, items, takes, fx instances...
This would be so great! Has anyone made a feature request yet?
spk77 is offline   Reply With Quote
Old 03-24-2018, 05:56 AM   #7
spk77
Human being with feelings
 
Join Date: Aug 2012
Location: Finland
Posts: 2,668
Default

Can we "attach" anything (values/tables etc.) to Lua userdata? I mean, is it possible to treat userdata as it was a Lua table? (I can't explain it better )
Maybe by using Lua metatables/methods. Too difficult for me, though.
spk77 is offline   Reply With Quote
Old 03-24-2018, 07:23 AM   #8
deeb
Human being with feelings
 
deeb's Avatar
 
Join Date: Feb 2017
Posts: 4,812
Default

@spk77: https://forum.cockos.com/showthread.php?t=196600
deeb is offline   Reply With Quote
Old 03-24-2018, 08:03 AM   #9
spk77
Human being with feelings
 
Join Date: Aug 2012
Location: Finland
Posts: 2,668
Default

Thanks deeb!
spk77 is offline   Reply With Quote
Old 03-24-2018, 09:02 AM   #10
nofish
Human being with feelings
 
nofish's Avatar
 
Join Date: Oct 2007
Location: home is where the heart is
Posts: 12,096
Default

Quote:
Originally Posted by spk77 View Post
Can we "attach" anything (values/tables etc.) to Lua userdata? I mean, is it possible to treat userdata as it was a Lua table? (I can't explain it better )
Maybe by using Lua metatables/methods. Too difficult for me, though.
I think the way to go would be table serialization (if I understand correctly), haven't done it myself though.

E.g.
https://forum.cockos.com/showthread.php?t=170820

https://forum.cockos.com/showthread.php?t=191009
nofish is offline   Reply With Quote
Old 03-24-2018, 09:38 AM   #11
spk77
Human being with feelings
 
Join Date: Aug 2012
Location: Finland
Posts: 2,668
Default

Thanks nofish!
By userdata, I mean "Lua userdata" (it's a Lua type), sorry for the confusion.

This is what I'd like to have
Code:
local tr = reaper.GetTrack(0,0) -- the type of tr is "userdata"
tr.tag = "Guitar"

This might be something related (I don't even know for sure) - see the test code/output at the end of this thread http://lua-users.org/lists/lua-l/2006-08/msg00245.html
spk77 is offline   Reply With Quote
Old 03-24-2018, 10:12 AM   #12
Xenakios
Human being with feelings
 
Xenakios's Avatar
 
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 8,062
Default

Quote:
Originally Posted by spk77 View Post
Code:
local tr = reaper.GetTrack(0,0) -- the type of tr is "userdata"
tr.tag = "Guitar"
In that, "tr" is light userdata, which is just a dumb raw C pointer without any properties that could get, set or added.
__________________
I am no longer part of the REAPER community. Please don't contact me with any REAPER-related issues.
Xenakios is offline   Reply With Quote
Old 03-24-2018, 10:15 AM   #13
nofish
Human being with feelings
 
nofish's Avatar
 
Join Date: Oct 2007
Location: home is where the heart is
Posts: 12,096
Default

Oh, misunderstood, sorry...
nofish is offline   Reply With Quote
Old 03-24-2018, 10:21 AM   #14
tack
Human being with feelings
 
tack's Avatar
 
Join Date: Jan 2014
Location: Ontario, Canada
Posts: 1,619
Default

I've a solution to this problem where I store serialized data to a custom JSFX using this hack. And then read it back and deserialize in the same way.

It'd definitely be nice to have native support in Reaper though, without the need for a custom JSFX.
tack is online now   Reply With Quote
Old 03-24-2018, 10:22 AM   #15
cfillion
Human being with feelings
 
cfillion's Avatar
 
Join Date: May 2015
Location: Québec, Canada
Posts: 4,937
Default

Quote:
Originally Posted by spk77 View Post
Can we "attach" anything (values/tables etc.) to Lua userdata? I mean, is it possible to treat userdata as it was a Lua table? (I can't explain it better )

This is what I'd like to have
Code:
local tr = reaper.GetTrack(0,0) -- the type of tr is "userdata"
tr.tag = "Guitar"
Code:
local tr = {ptr=reaper.GetTrack(0,0)}
tr.tag = "Guitar"
-- tr.ptr
cfillion is offline   Reply With Quote
Old 03-24-2018, 04:21 PM   #16
Xenakios
Human being with feelings
 
Xenakios's Avatar
 
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 8,062
Default

Quote:
Originally Posted by cfillion View Post
Code:
local tr = {ptr=reaper.GetTrack(0,0)}
tr.tag = "Guitar"
-- tr.ptr
This doesn't in any way solve the problem of keeping that data around in the Reaper project files etc, though.
__________________
I am no longer part of the REAPER community. Please don't contact me with any REAPER-related issues.
Xenakios is offline   Reply With Quote
Old 03-25-2018, 01:00 AM   #17
spk77
Human being with feelings
 
Join Date: Aug 2012
Location: Finland
Posts: 2,668
Default

Quote:
Originally Posted by Xenakios View Post
In that, "tr" is light userdata, which is just a dumb raw C pointer without any properties that could get, set or added.
I guess I mixed light userdata and full userdata.
  • full userdata is an object
  • light userdata represents a C pointer value


This is quite off-topic but interesting: (https://www.lua.org/pil/28.5.html)
28.5 – Light Userdata
The userdata that we have been using until now is called full userdata. Lua offers another kind of userdata, called light userdata.

A light userdatum is a value that represents a C pointer (that is, a void * value). Because it is a value, we do not create them (in the same way that we do not create numbers). To put a light userdatum into the stack, we use lua_pushlightuserdata:

void lua_pushlightuserdata (lua_State *L, void *p);
Despite their common name, light userdata are quite different from full userdata. Light userdata are not buffers, but single pointers. They have no metatables. Like numbers, light userdata do not need to be managed by the garbage collector (and are not).

Some people use light userdata as a cheap alternative to full userdata. This is not a typical use, however. First, with light userdata you have to manage memory by yourself, because they are not subject to garbage collection. Second, despite the name, full userdata are inexpensive, too. They add little overhead compared to a malloc for the given memory size.

The real use of light userdata comes from equality. As a full userdata is an object, it is only equal to itself. A light userdata, on the other hand, represents a C pointer value. As such, it is equal to any userdata that represents the same pointer. Therefore, we can use light userdata to find C objects inside Lua.

As a typical example, suppose we are implementing a binding between Lua and a Window system. In this binding, we use full userdata to represent windows. (Each userdatum may contain the whole window structure or only a pointer to a window created by the system.) When there is an event inside a window (e.g., a mouse click), the system calls a specific callback, identifying the window by its address. To pass the callback to Lua, we must find the userdata that represents the given window. To find this userdata, we can keep a table where the indices are light userdata with the window addresses and the values are the full userdata that represent the windows in Lua. Once we have a window address, we push it into the API stack as a light userdata and use the userdata as an index into that table. (Note that the table should have weak values. Otherwise, those full userdata would never be collected.)




Quote:
Originally Posted by cfillion View Post
Code:
local tr = {ptr=reaper.GetTrack(0,0)}
tr.tag = "Guitar"
-- tr.ptr
Thanks, that's how I'm doing it more or less at the moment.
spk77 is offline   Reply With Quote
Old 11-25-2021, 10:43 PM   #18
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

GetSetMediaTrackInfo_String() added in 6.36
MonkeyBars is offline   Reply With Quote
Old 11-26-2021, 12:30 AM   #19
X-Raym
Human being with feelings
 
X-Raym's Avatar
 
Join Date: Apr 2013
Location: France
Posts: 9,875
Default

@MonkeyBars
The function you mentioned isn't added in v6.36, it is a very old one.


What is new is extension prefix as parameters values.


Here is the changelog:
Code:
+ ReaScript: add P_EXT: prefix for extension-specific string state for GetSetMediaTrackInfo_String(), GetSetAutomationItemInfo_String(), etc     
+ ReaScript: allow using various GetSet..Value() functions with strings, automatically converting numbers to/from string
I never used this, not sure what the limitations if any, thx for having bring me that to my attention :P

Last edited by X-Raym; 11-26-2021 at 01:19 AM.
X-Raym is offline   Reply With Quote
Old 11-26-2021, 06:06 AM   #20
tack
Human being with feelings
 
tack's Avatar
 
Join Date: Jan 2014
Location: Ontario, Canada
Posts: 1,619
Default

I might have followed up on this thread and mentioned this. Yeah I've been using GetSetMediaTrackInfo_String() to store track metadata for some time in my dev branch of Reaticulate. It's been working great.

Code:
reaper.GetSetMediaTrackInfo_String(track, 'P_EXT:yourappnamehere', data, true)
Thanks for bumping, MonkeyBars.
tack is online now   Reply With Quote
Old 11-26-2021, 07:06 AM   #21
heda
Human being with feelings
 
heda's Avatar
 
Join Date: Jun 2012
Location: Spain
Posts: 7,239
Default

I gave up on using P_EXT on tracks because it was causing
performance issues when used on tracks with big VST chunks data.. like Kontakt etc...
Maybe now it works well. I may try it again.
heda is offline   Reply With Quote
Old 11-26-2021, 08:14 AM   #22
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Quote:
Originally Posted by X-Raym View Post
The function you mentioned isn't added in v6.36, it is a very old one. What is new is extension prefix as parameters values.
That's the important one for this thread!

Quote:
I never used this, not sure what the limitations if any, thx for having bring me that to my attention :P
I can't believe I turned X-Raym onto something he didn't know in ReaScript

Quote:
Originally Posted by tack View Post
I might have followed up on this thread and mentioned this. Yeah I've been using GetSetMediaTrackInfo_String() to store track metadata for some time in my dev branch of Reaticulate. It's been working great. Thanks for bumping, MonkeyBars.
That's awesome! Yes P_EXT is extremely useful, my pleasure!

Quote:
Originally Posted by heda View Post
I gave up on using P_EXT on tracks because it was causing
performance issues when used on tracks with big VST chunks data.. like Kontakt etc...
Maybe now it works well. I may try it again.
Oh no, that's too bad! Were you storing those large chunks of data in P_EXT or were they otherwise present?

I am planning to store a (pickled?) table of item GUIDs that could get pretty big on a large project, but likely nothing as large as FX chunk data I imagine (unless you had a million items or something... you never know with these game devs and other specialized pros).
MonkeyBars is offline   Reply With Quote
Old 11-26-2021, 10:08 AM   #23
tack
Human being with feelings
 
tack's Avatar
 
Join Date: Jan 2014
Location: Ontario, Canada
Posts: 1,619
Default

Quote:
Originally Posted by heda View Post
I gave up on using P_EXT on tracks because it was causing performance issues when used on tracks with big VST chunks data.. like Kontakt etc...
I haven't seen any obvious performance issues. I just did a quick test on a track with a few VSTis, including Kontakt with 10 patches (section patches from Spitfire Chamber Strings and Cinematic Studio Strings), with a track state chunk totaling about 15MB, and the call to GetSetMediaTrackInfo_String() takes around 15-30μs on my system. And there is no obvious scaling of this time with respect to the number of FX or Kontakt patches.

So if you give it another try and still run into the performance issue you observed, track chunk size might be a red herring, or perhaps you're talking about track states much larger than 15MB? (Or I've entirely misinterpreted the nature of the performance issue you had. )
tack is online now   Reply With Quote
Old 11-26-2021, 10:20 AM   #24
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Quote:
Originally Posted by tack View Post
I haven't seen any obvious performance issues. I just did a quick test on a track with a few VSTis, including Kontakt with 10 patches (section patches from Spitfire Chamber Strings and Cinematic Studio Strings), with a track state chunk totaling about 15MB, and the call to GetSetMediaTrackInfo_String() takes around 15-30μs on my system. And there is no obvious scaling of this time with respect to the number of FX or Kontakt patches.
Thanks for testing, tack!
MonkeyBars is offline   Reply With Quote
Old 11-26-2021, 10:25 AM   #25
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Since I've got you guys here, just wanted to point out my question on this other thread https://forum.cockos.com/showthread....78#post2500578

I want to save project data that gets saved to the Undo History, but it seems there's no API hook to do so – both Project ExtState and Project Info Strings/Data are outside Undo History.

I'd rather not have to create a hidden dummy track just to save my script data, but at this point I'm at a loss as to where I could place script data that gets reverted/restored on undo/redo. Any ideas, or is track P_EXT the best concession here?
MonkeyBars is offline   Reply With Quote
Old 11-26-2021, 10:50 AM   #26
tack
Human being with feelings
 
tack's Avatar
 
Join Date: Jan 2014
Location: Ontario, Canada
Posts: 1,619
Default

Quote:
Originally Posted by MonkeyBars View Post
I want to save project data that gets saved to the Undo History, but it seems there's no API hook to do so – both Project ExtState and Project Info Strings/Data are outside Undo History.
I am commonly fighting against REAPER's undo system when it comes to APIs. I wish it offered more control over around when and whether API calls would modify undo history.


Quote:
Originally Posted by MonkeyBars View Post
I'd rather not have to create a hidden dummy track just to save my script data, but at this point I'm at a loss as to where I could place script data that gets reverted/restored on undo/redo. Any ideas, or is track P_EXT the best concession here?
Could you store the P_EXT state on the master track?
tack is online now   Reply With Quote
Old 11-26-2021, 10:56 AM   #27
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Quote:
Originally Posted by tack View Post
Could you store the P_EXT state on the master track?
GREAT idea!! The master track is just track #0, right??
MonkeyBars is offline   Reply With Quote
Old 11-26-2021, 11:00 AM   #28
tack
Human being with feelings
 
tack's Avatar
 
Join Date: Jan 2014
Location: Ontario, Canada
Posts: 1,619
Default

Quote:
Originally Posted by MonkeyBars View Post
GREAT idea!! The master track is just track #0, right??
Call GetMasterTrack() to fetch the track object for the master track, and you can pass that to GetSetMediaTrackInfo_String().
tack is online now   Reply With Quote
Old 11-26-2021, 11:02 AM   #29
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Roger that! Thank you so much! This is definitely the solution.
MonkeyBars is offline   Reply With Quote
Old 11-26-2021, 04:16 PM   #30
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

I just did a series of tests and it appears using the Master Track to store data whose deltas are stored in Undo History is NOT an option after all! Darn.

Hopefully this chart helps other script devs. Final results:

SetProjExtState
OUTSIDE Undo History

Master Track GetSetMediaTrackInfo_String P_EXT
OUTSIDE Undo History

Regular Track GetSetMediaTrackInfo_String P_EXT
INSIDE Undo History

GetSetMediaItemInfo_String P_EXT
INSIDE Undo History

Surely safe to assume envelope and automation item P_EXT are also inside the Undo History.

My new FR here: https://forum.cockos.com/showthread....52#post2500852

Last edited by MonkeyBars; 11-26-2021 at 11:58 PM.
MonkeyBars is offline   Reply With Quote
Old 11-27-2021, 01:54 AM   #31
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Looks like Master Track Tempo Envelope changes get entered into undo points, so I'm using its P_EXT!
MonkeyBars is offline   Reply With Quote
Old 11-27-2021, 01:54 PM   #32
tack
Human being with feelings
 
tack's Avatar
 
Join Date: Jan 2014
Location: Ontario, Canada
Posts: 1,619
Default

Quote:
Originally Posted by MonkeyBars View Post
Looks like Master Track Tempo Envelope changes get entered into undo points, so I'm using its P_EXT!
Nice discovery. What modifies undo vs what doesn't feels a little ill conceived to me, so there's a risk this ends up being brittle. On the other hand, my intuition is that if something does change, it's likely to include master track updates via GetSetMediaTrackInfo_String() in undo history, rather than excluding tempo envelope.

Thanks for sharing!
tack is online now   Reply With Quote
Old 11-27-2021, 02:16 PM   #33
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Quote:
Originally Posted by tack View Post
Nice discovery. What modifies undo vs what doesn't feels a little ill conceived to me, so there's a risk this ends up being brittle. On the other hand, my intuition is that if something does change, it's likely to include master track updates via GetSetMediaTrackInfo_String() in undo history, rather than excluding tempo envelope.
Well put. That's also my worry, but then I am not able to deactivate the tempo envelope, so there's certainly no danger of the entire tempo envelope somehow getting eradicated.

Come to think of it, that's a bit of a UX glitch showing a box to deactivate the tempo env as if it were any other deactivtable env when it ain't.

My theory is that master track data is partially saved with project data (outside undo history), and partially as its own track (inside undo history).

Quote:
Thanks for sharing!
My pleasure sir! H/t @Julian for the idea
MonkeyBars is offline   Reply With Quote
Old 11-30-2021, 11:55 PM   #34
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Yep this is working great for my project
MonkeyBars is offline   Reply With Quote
Old 12-08-2021, 10:03 AM   #35
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Mespotine and I found some small API bugs that affect using this technique.

Last edited by MonkeyBars; 12-09-2021 at 12:21 AM.
MonkeyBars is offline   Reply With Quote
Old 12-09-2021, 12:21 AM   #36
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Turns out P_EXT was never officially supported for Master Track which explains its inconsistent behavior.

However, schwa is working on it for the next release! Actually, in the latest dev version, changes in the Master Track (non-env) P_EXT are now being saved in undo points.

Last edited by MonkeyBars; 08-30-2022 at 09:28 PM.
MonkeyBars is offline   Reply With Quote
Old 01-19-2022, 05:07 PM   #37
MonkeyBars
Human being with feelings
 
MonkeyBars's Avatar
 
Join Date: Feb 2016
Location: Hollyweird
Posts: 2,630
Default

Master Track P_EXT support was officially added in v6.43.
MonkeyBars is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 07:09 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.