I had this same issue. I discovered that the Waves plugins were not yet Retina compatible. Waves has since issued an update. If you've never updated Waves on a Mac before, it can be a little tricky.
I found that the best method was to uninstall my Waves plugins, using the Waves Uninstaller (located at "Macintosh HD/Applications/Waves/Utilities/Waves Uninstaller).
After uninstalling, I updated the Waves installer (which can be downloaded at
http://www.waves.com/content.aspx?id=96).
After updating the installer, I opened new installer, selected my plugins to install, and ran the installer.
After running the installer, all of my plugins were up to date, and the oversized font issue was resolved.