Setting up the audio device on OSX is described in section 1.12.2 (page 22/23) of the User Guide (please get it here if you don't have it yet, it's absolutely essential:
http://www.cockos.com/reaper/userguide.php ) - section 3.5. on page 55 shows you how to set up a track for recording.