Converting the video is a good idea. I completely recommend
TEncoder, converting to the ProRes Proxy codec in a Quicktime or MKV container. You can even convert the contained audio to WAV or Flac with this by picking "none" for the video codec and WAV or Flac for the audio part. Check out the application. It's very useful.
If you have not yet done so, install Video Lan Client, i.e. VLC.
Make sure to use the Windows 64-bit version for the Reaper 64-bit Windows version. 32->32. OSX/Macos only has one version.
That should cover most of your playback needs.
Check the SOURCE PROPERTIES of the video item to make sure it is using VLC as a playback engine. CTRL/CMD+F2 is the default shortcut.