VST2s on Linux sorted as effects instead of instruments

[Tas from planet u-he]

Tested with UbuntuStudio 21.04 and Reaper 6.81.
Can be tested with the demo versions of Zebra2 or Repro.

The VST2 versions of plugins like Filterscape, Repro and Zebra2 will wrongly be classified as effect plugins rather than instruments. This happens with all u-he plugins that contain more than one plugin.

The VST3 and CLAP versions are sorted correctly.
And the problem does not exist in the Mac and Windows versions of Reaper, only happens on Linux.

[audiojunkie]

Originally Posted by Tas from planet u-he

Tested with UbuntuStudio 21.04 and Reaper 6.81.
Can be tested with the demo versions of Zebra2 or Repro.

The VST2 versions of plugins like Filterscape, Repro and Zebra2 will wrongly be classified as effect plugins rather than instruments. This happens with all u-he plugins that contain more than one plugin.

The VST3 and CLAP versions are sorted correctly.
And the problem does not exist in the Mac and Windows versions of Reaper, only happens on Linux.

@justin @schwa This is the U-he developer of the linux version of their products. There has been finger pointing back and forth about this issue. Hopefully now, you can both start a correspondence between each other an get the issue resolved.

Please comment and work this issue out together so that there is an agreement between both parties that result in a solution that works for us customers, regardless of whose application the bug lies! 🙂

Please!!

[Justin]

It seems these plug-ins, after we call effShellGetNextPlugin, do not update their return value for effGetPlugCategory, nor do they update their flags variable. E.g. effGetPlugCategory still returns kPlugCategShell, and flags is still 0. If either of those values can be updated in response to effShellGetNextPlugin, that would fix. Or if u-he has a suggestion for another way to query whether the currently indexed plug-in type is a VSTi, we'd like to hear it.

The handling of this is identical on our end on Linux as it is on Win/macOS, so it's likely different on the u-he Linux side for some reason?

[audiojunkie]

Originally Posted by Justin

It seems these plug-ins, after we call effShellGetNextPlugin, do not update their return value for effGetPlugCategory, nor do they update their flags variable. E.g. effGetPlugCategory still returns kPlugCategShell, and flags is still 0. If either of those values can be updated in response to effShellGetNextPlugin, that would fix. Or if u-he has a suggestion for another way to query whether the currently indexed plug-in type is a VSTi, we'd like to hear it.

The handling of this is identical on our end on Linux as it is on Win/macOS, so it's likely different on the u-he Linux side for some reason?

Thank you, Justin for responding!

[Tas from planet u-he]

Thanks for the hints. We looked into this some more, and noticed that there seems to be another undocumented method to get the plugins categorized besides the method we had implemented.
This might be the reason why it works in any other Linux host, but not in Reaper.
On Mac and Win, we do not house the VST2 plugins in a single shell, so that might explain why it works ok on Mac and Win, but not on Linux.
We'll try to make it work in Reaper on Linux now.

[Reaktor:[Dave]]

Originally Posted by Justin

It seems these plug-ins, after we call effShellGetNextPlugin, do not update their return value for effGetPlugCategory, nor do they update their flags variable. E.g. effGetPlugCategory still returns kPlugCategShell, and flags is still 0. If either of those values can be updated in response to effShellGetNextPlugin, that would fix. Or if u-he has a suggestion for another way to query whether the currently indexed plug-in type is a VSTi, we'd like to hear it.

The handling of this is identical on our end on Linux as it is on Win/macOS, so it's likely different on the u-he Linux side for some reason?

Thanks for looking into this, Justin. For backwards compatibility reasons, the VST2 Shell feature was not enabled on macOS and Windows. The initial plugin release of the Linux version meant there was nothing to be backwards compatible with, so Linux builds are made with the VST Shell feature enabled.

I tried to debug the scanning process but I'm not hitting any breakpoints (I guess REAPER tries to avoid crashing due to faulty plugin scans so it spawns the plugins in separate processes). Meaning I can only place some logs and try to piece it together from that.

When comparing the logs of Reaper's plugin scan with the logs from Bitwig (which does pick up the correct categories), there seems to be a difference in the way the calls are made.
This is how the log looks when Bitwig scans Zebra2:

Should create effect instance with audioMasterCurrentId 2 and our plugId 0
creating shell plugin
VST_Shell::getPlugCategory: Returning category kPlugCategShell!
VST_Shell::getNextShellPlugin: Returning plugin_id 1397572658
VST_Shell::getNextShellPlugin: Returning plugin_id 1397572659
VST_Shell::getNextShellPlugin: Returning plugin_id 1919243824
VST_Shell::getNextShellPlugin: Returning plugin_id 1397578034
Should create effect instance with audioMasterCurrentId 2 and our plugId 1397572658
creating plugin id: 1397572658
Should create effect instance with audioMasterCurrentId 2 and our plugId 1397572659
creating plugin id: 1397572659
Should create effect instance with audioMasterCurrentId 2 and our plugId 1919243824
creating plugin id: 1919243824
Should create effect instance with audioMasterCurrentId 2 and our plugId 1397578034
creating plugin id: 1397578034

This is how the log looks when Reaper is scanning Zebra2:

Should create effect instance with audioMasterCurrentId 2 and our plugId 0
creating shell plugin
VST_Shell::getPlugCategory: Returning category kPlugCategShell!
VST_Shell::getNextShellPlugin: Returning plugin_id 1397572658
VST_Shell::getPlugCategory: Returning category kPlugCategShell!
VST_Shell::getNextShellPlugin: Returning plugin_id 1397572659
VST_Shell::getPlugCategory: Returning category kPlugCategShell!
VST_Shell::getNextShellPlugin: Returning plugin_id 1919243824
VST_Shell::getPlugCategory: Returning category kPlugCategShell!
VST_Shell::getNextShellPlugin: Returning plugin_id 1397578034
VST_Shell::getPlugCategory: Returning category kPlugCategShell!

From the plugin's side, it looks like REAPER is expecting the plugin to change its plugin_id and other stuff after it received the getNextShellPlugin opcode. Bitwig seems to call VSTPluginMain with all the names provided by the plugin via getNextShellPlugin and that gets them the correct metadata.

The VST2 header file "audioeffectx" contains a comment for "getNextShellPlugin" stating that its only intention is to return the next plugin name (implying that the last quered index needs to be remembered internally, so hosts can call this until they have all plugin names). It does not mention that the plugin should be prepared for other calls like getPlugCategory to change their output accordingly. Is REAPER's behavior common practice with the VST2 Shell?

I made some changes to the VST2 shell implementation so that the plugins should report the corresponding category as well as the updated flags after calling getNextShellPlugin(). With that change, REAPER is picking up the correct category over her just fine. This also doesn't seem to break the plugin scanning in Bitwig, so even if that behavior is not strictly following the VST2 header comments, it seems to be working.

[Justin]

shrug, one of those vague parts of the spec... other shell plug-ins do this, letting us avoid instantiating every type to query the state... I guess we could support the full init too, eventually.

[audiojunkie]

Thank you, all of you, for working this out!! I really appreciate it!! 🙂

[schwa]

Originally Posted by Reaktor:[Dave]

I tried to debug the scanning process but I'm not hitting any breakpoints (I guess REAPER tries to avoid crashing due to faulty plugin scans so it spawns the plugins in separate processes). Meaning I can only place some logs and try to piece it together from that.

Hidden REAPER config: in REAPER.ini, add the line vstscan=1 (or if a vstscan= line already exists, add 1 to the value) and restart REAPER, and then plugin scanning will be in-process which makes it easier to debug scan-time problems. Although of course scan-time crashes will then bring down REAPER which is why there's no user-facing UI to set this config value.

If needed, you can find the path to REAPER.ini via the action "Show REAPER resource path in explorer".

[Reaktor:[Dave]]

Originally Posted by schwa

Hidden REAPER config: in REAPER.ini, add the line vstscan=1 (or if a vstscan= line already exists, add 1 to the value) and restart REAPER, and then plugin scanning will be in-process which makes it easier to debug scan-time problems. Although of course scan-time crashes will then bring down REAPER which is why there's no user-facing UI to set this config value.

If needed, you can find the path to REAPER.ini via the action "Show REAPER resource path in explorer".

Thank you so much for mentioning! As I primarily use REAPER to load our plugins when developing, this is absolutely great to know.

[Tale]

Originally Posted by schwa

Hidden REAPER config: in REAPER.ini, add the line vstscan=1 (or if a vstscan= line already exists, add 1 to the value) and restart REAPER, and then plugin scanning will be in-process which makes it easier to debug scan-time problems.

Cool! Under which section should I add this, under [REAPER]?

[schwa]

Originally Posted by Tale

Cool! Under which section should I add this, under [REAPER]?

yep

[Tale]

Is is vstscan=1 or vst_scan=1? Because the reaper.ini on my Mac already contained vst_scan=0, so I'm guessing it should actually be vst_scan=1.