Reaper 6.14 x64
Windows 10 x64
Video showing how to reproduce
Say you have +50 ms playback time offset on a track to sync it with the other instruments in the project. Then you want to freeze the track. The playback offset will be baked into the frozen audio, but the playback offset in the track itself is preserved.
This means that your audio now has +100 ms playback offset in total. You have to go into the track IO settings and uncheck playback offset to have the correct amount.
I think Reaper should either uncheck the box after freezing, or it should disregard playback offset when rendering the frozen audio.