Ok, I can confirm this but it's tricky!
Also using a magic mouse...
Some notes:
- It only happens in the first column while in columns mode.
- Using an emulated smooth scroll does have the same issue.
- After you successfully scroll once, the dialog works properly even if you restart Reaper!
- Scrolling with a wheeled mouse fixes it for the smooth scrolling mouse.
- Having the window selected or not does not change it.
It's rather difficult to reproduce this and I've only seen this happen in Reaper.