Old 02-15-2010, 06:59 AM   #1
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default RPP chunks issues: hangs, CPU spikes, crashes, ..

This thread deals with some issues about GetSetObjectState (or its "children" for ReaScript) that allows reading/patching RPP chunks.

FACTS:

1) Infinite hangs (not to say "deadlocks".. but 0% CPU in those cases..)
- Calling GetSetObjectState in a multi-threading context leads to infinite hangs
- This occurs even when simply "getting" chunks
- This is more reproductible when playing and this is even worse when recording
EDIT: FIXED/IMPLEMENTED (v3.31), some few hangs remain, see bellow.

- Calling GetSetObjectState through a standard action itself called from a thread leads to the same hangs
[edit]I have to test that one in v3.31[/edit]

- Calling GetSetObjectState through a through an audio hook leads to the same hangs
EDIT: FIXED/IMPLEMENTED (v3.31) may glitch, see schwa's posts 11 & 15.

- Calling GetSetObjectState through a standard action may seem a bit safer, but no: same hangs can be monitored when recording/playing. "may" seem a bit safer, 'cause, for ex, patching FXs bypass while REPAER plays is ok from an action but the *exact same code* called thru a thread will randomely hang. however, for ex, patching FXs on/offline while REPAER records will hang in both cases.
EDIT: Since v3.31, GetSetObjectState "seems" threadsafe. From the main thread, some actions can hang when recording (see post #16), in my case, I "fixed" that by preventing chunk patching during record (i.e. !(GetPlayState() & 4)). Also, not sure that patching something that's being recorded makes sense..

Note: I more than confirm that other API functions (that do not process chunks) are thread-safe, never saw unpredictable behavior or hangs! I can post some code/plug/extension showing those issues.

2) Patching chunks => CPU spikes, serious hangs
When patching chunks, I'm able to monitor REAPER hanging up to 40s with a CPU spike of the same duration!!
IMHO, a real show stopper.
It's a known issue, I've read that it depends on the kind of VSTs defined on the patched tracks. I don't think so: for ex, a huge Drums VSTi will indeed stuck REAPER for a few seconds, but the way I can stuck REAPER for 40s is rather due to massive in-project MIDI items or enveloppes..
EDIT: forget that, apologizes: show stopper in my code, yeah! There's a bunch of string processing behind the scene, I handle that much better now..

3) crashes
The following crash examples can be triggered through standard REAPER actions GetSetObjectState-based (i.e. no threads)
- Patch when updating the track concurently: crashes
Ex: patching a track while the user adds an FX, may be ok but rather *randomely* crash
- Patch a track beeing record/played: random poofs
- Patch from a thread may hang REAPER (as said above) but may also crash REAPER and may also work
I can post some offsets, but not sure they'll be usefull (ModName: "unknown", "ntdll.dll")
[edit] very random: I still looking for a reliable repro about that. Some crash offsets examples attached in post 8
EDIT: Part of these issues were FIXED in v3.31. AFAIK, all the other remaining random crashes can be fixed by removing all ids before patching. Those crashes become less random when you patch like a "machine-gun" (i.e. the repro I was looking for, above). I found that fix the very hard way, I don't know if it's a bug or not..

4) Minor: Ignored patch, GUI update issues
Here, I think those issues mean that what I try to do is simply out of the scope of GetSetObjectState
Examples:
- Unfloating a FX window is ignored (showing it is OK)
=> EDIT: dedicated API functions has been added for that in the meantime

- Patching the ME conf is ignored
- Removing/adding/updating sends does not update all GUIs: routing matrix, the mixer and brand new TCP i/o icons,.. EDIT: I entered a bug for this one, it's here
For the GUI refresh issues, TrackList_AdjustWindows(), UpdateTimeline() and stuff don't help. EDIT: just some examples, there are other cases of refresh issues.

Last edited by Jeffos; 08-23-2010 at 02:38 PM. Reason: New colorful status, à la Airon!
Jeffos is offline   Reply With Quote
Old 02-15-2010, 07:31 AM   #2
MikeLacey
Human being with feelings
 
Join Date: Dec 2006
Location: UK
Posts: 789
Default

Hi Jeff,

I've not come across these problems myself using ReaScript - have you had issues using Perl or Python?
__________________
Mike Lacey, Leicestershire, UK
MikeLacey is offline   Reply With Quote
Old 02-15-2010, 08:16 AM   #3
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

hi Mike! are you sure ? can you do this test: make a script that toggles something in a FX chain with a track of 40 min of *true* MIDI data with, say 8 FXs including a big one (EZD or AD). Shortcut your script to a key, play, press the key like crazy. hang ?

Now, make a script that infinively loops the one above => you're ok for 16000 iterations if I remember right. Run it on a track with MIDI data (but less than 40 min). Arm the track, press *record*, and now, while the script is being executed, do some wicked things on that track: remove/add some FXs, bypass, offline FXs, remove anticipative processing, remove/add receives (not sends, patched elsewhere). hang/crash ?
Jeffos is offline   Reply With Quote
Old 02-15-2010, 08:44 AM   #4
MikeLacey
Human being with feelings
 
Join Date: Dec 2006
Location: UK
Posts: 789
Default

Eeek!
__________________
Mike Lacey, Leicestershire, UK
MikeLacey is offline   Reply With Quote
Old 02-15-2010, 11:39 AM   #5
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

Eeek !?? It reminds me when I first heard "Nuff said".. I wanted to ask "but who is nuff ?" ha! ha!

The OP is a list of facts. No analysis, no suggestions, just facts.
I'm quiet sure some stuff is just "not allowed" or would need other API tools, mutex things,.. but we have no way to know that, so it's an issue itself. Also, I've tracked programming errors, did some serious stressing tests, dig enough into the API before saying that.

Pita! There was a crazy potential here (really crazy!!)

______________

Just in case, for repro in REAPER:

I quiclky made a VST POI ("Proof Of Issue") that would have been interesting for live perfs, attached here: it automatically disables FXs (i.e. bypass or offline) -except itself- when no midi note has been received after a configurable timeout and re-enables them since a note is received. Also, when it receives a pg change x, it'll remove all other tracks receives and automatically create a new send from the track where this VST resides to the track x (i.e. the pg change).
Tech. details: for bypass/offline, when the related param(s) is/are enabled, this VST polls bypass and/or offline states through threaded calls to GetSetObjectState(), READ ONLY. Poll freq. = 500 ms. For automatic send creation, it does several threaded calls to GetSetObjectState(), WRITE (no polling).

Threading "infinite" hangs:
=> Random hang: add a bunch of other FXs, enable the auto bypass param, loop play, wait... all other FXs should be automatically disabled, but, given a random point REAPER will hang. Repetitively running the *same code* as an action is ok..
=> If you're in a hurry: do the same with the auto offline parameter, it will hang very quickly

Threading "infinite" hangs or crash:
=> record, then send a pg change to this VST and pray..

Standard action ""infinite" hang:
=> record a track with a bunch of FXs, while recording, run a "Toggle all FX offline/online (except FX y)" that I've put in SWS extensions => almost sytematic hang. That's also why I'll soon remove those actions from the open source, but the question is, shouldn't I rather remove all objectState based extensions ?

Last edited by Jeffos; 02-18-2010 at 01:52 PM. Reason: zip removed (quota!)
Jeffos is offline   Reply With Quote
Old 02-15-2010, 12:54 PM   #6
sfzgeek
Human being with feelings
 
sfzgeek's Avatar
 
Join Date: Feb 2009
Location: Dunedin, New Zealand
Posts: 205
Default

Cool. Jeffos is trying to blow things up. I'll check out these "chunky" posts in detail when I get a quiet moment...
__________________
My rawk band: The Hidden Venture.
sfzgeek is offline   Reply With Quote
Old 02-15-2010, 02:02 PM   #7
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

then, while you (really) are in the snow, an australian guy with *massive* sunglasses chimes in and says:
Quote:
Originally Posted by sfzgeek View Post
Cool.
HA! ha! I love this place!

I edited the titles of the OP: were a bit agressives, as as said it elsewhere I like the API, the OP is meant to report issues and to try to help to improve that.
Also, for some helpfull people that are trying some betas I did: no issue with them !!
Jeffos is offline   Reply With Quote
Old 02-16-2010, 11:06 AM   #8
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

Dale, I forgot: it already smells like another epic fail but thanks for your support! And yes, please, when you'll have a moment, let us know if/how much you can stuck REAPER from ReaScript.

But I'm surprised no one else says nothing..

So, I go ahead: of course, these issues are common to all ObjectState extensions (not only Jeffos' ones), other example:
SWS snapshots.

If you want reproductions steps for the inifinite hang issue and/or if you want to see REAPER hanging for 1 minute with a CPU spike of the same duration using snapshots, just let me know. Note: native pb, there's nothing Tim can do to prevent those issues.

Some exotic crash offsets attached too (triggered in a multi-threading context if I remember right).
Attached Files
File Type: zip chunk_patching_crash_offset.zip (270.6 KB, 274 views)
Jeffos is offline   Reply With Quote
Old 02-16-2010, 12:07 PM   #9
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,749
Default

Thank you for all the detail.

GetSetObjectState should only be called from the main thread. We'll note this in the API header. The only way Reaper could safely protect against concurrent API callers would be with a global lock on everything. The best solution is to treat GetSetObjectState like a GUI activity, and require it to happen in the main thread. Note a similar comment about threading here http://www.cockos.com/reaper/sdk/vst/vst_ext.php with regard to VST effSetChunk.

There's a bit of information overload in your post, would you mind deciding which point you think is the most critical and repeating that one? After we address it, we can move on to the next one.
schwa is offline   Reply With Quote
Old 02-17-2010, 01:34 PM   #10
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

first: many thanks!

You're right: there's bunch of things here. Tho, one was recently dropped from this list in v3.22: thanks for that too.

So, ok, I'll take them in the OP order. The 1st point deals with threading (here I talk about VST context but I guess it could also be interesting for CSurf too).

To be honest, I still brainstorming what you said: I quiclky understood I had to require it to happen in the main thread, the pb is I don't know how to do it (in a clean way). I guess things like On_MainCommand combined with home-made actions are what you're talking about (?) but, then, how get I get the result ? Also, I don't really get the comparison with effSetChunk (and I don't think you're advising me to do the job there as it's called very rarely).

schwa, if you see a way/workaround to do the job in a thread safe way, please can you re-give me that clue with the assumptions that: 1) I'm french 2) I code VSTs (well.. IPlugs ) only since last october (and from time to time).

Next point: infinite hangs.
I'll only focus on the main true issues with as clear as possible repro.
Jeffos is offline   Reply With Quote
Old 02-17-2010, 01:41 PM   #11
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,749
Default

For the threading issue, what we can do is inside GetSetObjectState, Reaper can check if it's being called from the main thread, and if not, lock everything while processing the call. If you call GetSetObjectState from an audio processing worker thread, this may cause audio to be interrupted, but that's life.

If you have a plugin that wants to call GetSetObjectState and you want to be sure you are calling from the main thread, make the call from the graphics handling code. If you are making the call in response to a user pressing a button, you are already in the main thread. If you want to make the call in response to an audio signal, you have to do something more involved, like setting up a timer in the graphics thread, and setting a flag from the audio thread.
schwa is offline   Reply With Quote
Old 02-18-2010, 01:06 AM   #12
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

You're going way too fast for me!! I just reply to the above post, I'll post the hangs repro later this evening..

About the "global lock" proposition: dunno. On one hand, we can assume that a guy doing some threading has some coding knowledge and thus, will quickly caught GetSetObjectState isn't thread-safe, but, on the other hand, generally speaking, interupting audio is better than hanging/crashing also, I think it could make sense 'cause when you patch it means you're doing some serious things (not just changing a volume, for ex)
So, I'd say "up to you!" If it's easy.. that would definitively fix the issue #1
(I don't ask for it very hard as it seems I'm the only guy to do that kind of things.. for the moment..)

About ObjectStating from VSTs: OK! got you 5/5 this time! Silly Q: I could have asked this one in the WDL forum, sorry, I simply didn't know that.. I did a quick test: the "Proof of issue" attached above turned into a "Proof of concept", calling GetSetObjectState in a thread-safe way form a VST is possible, woo! This gonna radically simplify our live set-up: many, many thanks!

[big fan] schwa, in case you're not already aware of that: YOU ROCK!!!! [/big fan]
Jeffos is offline   Reply With Quote
Old 02-18-2010, 01:47 PM   #13
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default "Infinite hangs" (smells like deadlock, no CPU use)

This seems to occur when playing/recording and patching things concurently. It also depends on the patched things (some are ok). Quite easy to reproduce when recording.
Please use the sws_extensions custom build attached here (win32, for Cockos tests only).

Test case 1: "BYPASS" patch
- add 8 random FXs including AD or EZD as 8th FX (not related to the issue) on an empty track x
- arm the track x (midi or audio), record
- while recording, run the action "SWS/S&M: Toggle FX(x) online/offline (except FX 1)"
=> hang (if not, re-toggle, re-toggle, ... should hang very quickly)

Test case 2: a bit similar
- Enable FX Chains as part of SWS snapshots (original code in that build)
- add 8 random FXs including AD or EZD as 8th FX on a track x
- select track x, run the action "Save as snapshot 1"
- put all FXs offline for track x (with the native action which is safe ;-)
- select track x, run the action "Save snapshot 2"
- select track x, run the action "Recall snapshot 1" and assign it to a key, say "A"
- select track x, run the action "Recall snapshot 2" and assign to a key, say "B"
- arm the track x (midi or audio), record
- while recording, press B
=> should hang here, otherwise A,B,A,..
note: with snapshots, I've also seen one (but only one) hang without playing/recording

I can imagine what happens behind the scene when off/onlining..
Do we simply have to check the play/record state before patching? is it allowed? a bug?

Test case 3: now, we patch something else, for the ex. Both chunks "LASTSEL" et "SHOW" patched in one go here
- add 8 random FXs including AD or EZD as 8th FX on an empty track x
- arm the track x (midi or audio), record
- while recording, run the action "SWS/S&M: Show FX chain (FX 2) for selected track(s)"
=> should hang here

schwa, if it can help, what I can do is to try to patch things and report which ones suck (not all, 'cause I need to know what I patch ;-). Not much work (2 lines to patch) but not before at least one week, would it be usefull ?

[edit] I forgot! In all repros above, track x is empty, no items.
Attached Files
File Type: zip reaper_sws.zip (355.2 KB, 211 views)

Last edited by Jeffos; 02-25-2010 at 04:19 AM. Reason: zip updated...
Jeffos is offline   Reply With Quote
Old 02-19-2010, 04:38 AM   #14
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

I forgot: no threading in the repros above, I've moved on "1bis) infinite hangs with standard actions".

Quote:
Originally Posted by Jeffos View Post
Please use the sws_extensions custom build attached here (win32, for Cockos tests only).
well, if someone wants to confirm, he/she's of course welcome (in the crazy case these issues only occur on my set-up..). For test cases 1 & 2, a standard SWS extensions build can be used.
Jeffos is offline   Reply With Quote
Old 02-20-2010, 06:06 AM   #15
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,749
Default

The official Reaper 3.31 release has more complete thread locking when the API calls GetSetObjectState, as explained above this can cause audio glitches when called from an audio worker thread, but I can't get your examples to hang anymore with 3.31.
schwa is offline   Reply With Quote
Old 02-20-2010, 02:48 PM   #16
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

I still able to reproduce all 3 hangs repro of post #13 in v3.31 final (i.e. using standard actions while recording, while playing seems ok).
That said, GetSetObjectState now looks "thread-safe", e.g. the sucky plug I posted above now works in v3.31, even while recording, also couldn't hear a glicth while patching+playing and getting seems seamless!!
Sorry, but I cannot continue to thank you in each post I do, please see my new sig.

In other words, now, it seems safer to call GetSetObjectState from a thread that from standard action (!#çù%!!??), the exact same code peformed in a thread is ok but can hang when called as an action, e.g. plug vs entension posted above when off/onlining.
Jeffos is offline   Reply With Quote
Old 02-21-2010, 02:43 AM   #17
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

ok, I did some others wicked tests: the whole thing seems way much safer, couldn't trigger a single crash! Also, I let REAPER looping all night long with a thread getting a chunk every 500ms. This morning REAPER was still healthy and looping. Please, see my sig.

[edit] too bad: re-did the test (2/22), this morning REAPER was hang.. The only diff was that the looped/polled track was also armed.

So, the only remaining hang case I saw is when we patch *some* stuff with standard actions *while recording*.

I edited fixed/implemented things the OP.

So, let's not be maniac here: may be we can simply prevent patches while recording form our extensions code ? (moreover, I think it can make sense..) and move on next issue ?

Last edited by Jeffos; 02-22-2010 at 12:55 AM. Reason: super random hang when threading..
Jeffos is offline   Reply With Quote
Old 02-23-2010, 12:43 AM   #18
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default random threading hang

Quote:
Originally Posted by Jeffos View Post
[edit] too bad: re-did the test (2/22), this morning REAPER was hang.. The only diff was that the looped/polled track was also armed.
ok, re-re-re-re-did that threading endurance test, I think I've nailed the hang. It sounds weird, but this is just what the tests tend to show: those hangs only seem to occur when REAPER is minimized/doesn't have focus.

repro:
- add 8 random FXs including AD or EZD as 8th FX
- add the test VST to the chain (just in case: https://stash.reaper.fm/4781/AutoTrackReaControl.zip)
- enable its "auto bypass" parameter => the thread starts to poll
- add a random audio file on that track + arm it
- make a loop, play
- MINIMIZE REAPER (and do something else..)
=> should hang in a few minutes

note: AFAIK, when REAPER has the focus, the threading endurance test is always OK (test of 8-10 hours).

HTH.

Last edited by Jeffos; 02-26-2010 at 04:21 AM. Reason: updated link
Jeffos is offline   Reply With Quote
Old 02-25-2010, 06:06 AM   #19
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 15,749
Default

So ... where are we with these bugs? What's the highest priority problem you see at the moment?
schwa is offline   Reply With Quote
Old 02-25-2010, 07:50 AM   #20
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

thanks again for taking care of those issues.
Quote:
Originally Posted by schwa View Post
So ... where are we with these bugs?
I can reproduce both the concurrent patch/record issue (just did the test1 of post #13) and the remaining threading hang (post #18, though I had to wait 20min or so) in v3.32pre2.
Quote:
Originally Posted by schwa View Post
What's the highest priority problem you see at the moment?
the remaining threading hang (we can avoid the recording one from extensions and perharps even live with it (?): a bit like undos forbidden while recording).
Jeffos is offline   Reply With Quote
Old 02-26-2010, 04:20 AM   #21
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

po po po! I see some downloads for the test VST plug!!
Please, note that even if it *seems* to work, it's really not efficient/recommended. This is for test purpouse only!. I'll post a clean version when thoses issues will be sorted out.

Also, I detail what it does:
- Polls a track chunk every 500ms (since bypass and/or offline param(s) are enabled)
- But it also writes a track chunk every configured timeout seconds (since bypass and/or offline param(s) are enabled), even if it doesn't need to!
- Both actions are threaded (i.e. not done in the main thread).

Also, there was a bug (*), updated version here:
https://stash.reaper.fm/4781/AutoTrackReaControl.zip

(*) was coded in "blind mode" before v3.31.. of course, it's not my style to code some bugs
Jeffos is offline   Reply With Quote
Old 08-06-2010, 09:50 AM   #22
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

OP updated.

In short, there's no more show stopper. Thank you Cockos. The remaining issues can be prevented from 3rd party code (random crashes, remaining hangs).
Also, I'm doing some benchmarks using HUGE chunks these days, it seems that the set is faster too. I have to downgrade to confirm that..
[EDIT] benchmarks didn't proved anything, the issue was rather on my side..

Last edited by Jeffos; 08-23-2010 at 02:39 PM.
Jeffos is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 09:45 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.