Reaper tries to remove the plugins IPlugView, before closing VST3 context menus

[dylan.w]

Linux, Reaper 6.43

When closing the FX window while a VST3 context menu is open, the context menu should be closed first before trying to remove the plugin's IPlugView.

This causes problems when using Reaper with VST bridging software (Yabridge). The above description came from Robbert (the developer of Yabridge), after investigating why Reaper froze when I attempted to close a plugin with context menus open.

[Fergler]

Maybe this is why the preset dropdown in surge synth remains open if you close the plugin?

[robbert-vdh]

For the record, what happens when you close an editor window while a context menu is open is that during the IRunLoop:nFDIsSet() call where the plugin calls IContextMenu:opup(), REAPER calls IPlugView::removed() and IPlugView::release() from a nested event loop. This means that the plugin's plugview will be deallocated while it is still in the middle of calling IContextMenu:opup() further down the call stack on the same GUI thread. This can lead to all kinds of fun UB.

[dylan.w]

Thanks for adding the additional info, Robbert.

I'm now experiencing this bug (or at least it seems similar from user perspective) without any bridging or 3rd party plugins involved. I couldn't be 100% sure if it's the exact same thing, so I've made a new bug report here: https://forum.cockos.com/showthread.php?t=261548

[robbert-vdh]

Yes, it does not have anything to do with bridging. Any plugin would suffer from the same problem (on Linux at least, don't know how this behavior is implemented on Windows), although with some it may not be apparent. The main issue is that objects are being destroyed while in the middle of interacting with that object. In theory this issue could be avoided on the plugin's side by creating another copy of the smart pointer for the lifetime of the event handler so it can't get dropped until the function returns, but that should not be necessary and no plugin would normally do that.

[schwa]

How are you closing the plugin window specifically?

[robbert-vdh]

This happens when just clicking on the close button in KWin's window decoration while the context menu is open. Maybe this backtrace better demonstrates what's going on:



The cyan parts are REAPER's event loop. The green part is a plugin's IRunLoop:nFDIsSet() handler that REAPER calls when something has been written to the plugin's socket (when the wants to do something from REAPER's GUI thread). From that event handler the plugin calls REAPER's IContextMenu:opup() implementation. There's no symbol for that, but that's frame #20. At that point REAPER's GUI event loop takes over to handle the context menu. Then when you try to close the X11 window while all of this is going on, REAPER tries to call IPlugView::removed() in the topmost cyan part followed by IPlugView::release(), which ends up deallocating the IPlugView implementation. This is the yellow part at top of the call stack. Since the IRunLoop implementation and IPlugView implementation are canonically implemented by the same object, this will cause some very fun undefined behavior unless the plugin is specifically designed to work around this.

[schwa]

OK thanks. It may be that this behavior is specific to this window manager. I think that Windows and Mac will close a currently open context menu before closing a window. We'll think about how we might be able to mitigate this on our side.

[Justin]

Yeah I think for now we'll make swell-gdk ignore those clicks if a menu is open (instead just closing the menu) -- this is what macOS does anyway -- eventually we can have it defer the close action until after the menu has closed and normal execution has resumed.

[EvilDragon]

Originally Posted by Fergler

Maybe this is why the preset dropdown in surge synth remains open if you close the plugin?

Hold on, I don't see this happening on Windows. Which version of Surge, and which plugin flavor?