1) Run Reaper on Mac
2) Open FX on any track
3) Add AUSampler plugin (comes with every Mac)
4) Click the little piano button at the bottom right of the plugin UI
Before:
https://i.imgur.com/m2PpQ9W.png
After:
https://i.imgur.com/UOXmuSl.png
The top half of the plugin UI gets cut off, now you can only interact with the bottom half. Scrolling up doesn't work. Changing settings for floating windows doesn't help.
I've tried the same plugin in GarageBand and AULab, there it resizes itself fine and the problem doesn't happen.