The main problem I have with JS is its awful syntax. I've programmed in 20+ languages and it's syntax is the most odd looking I've seen.
It probably wouldn't be all that bad but the editor is horrendous. Like to use shift-ins to paste and all that ends up doing is toggling ins and I end up overwriting text unexpectedly. It has other weird behavior that makes me queasy using it. In fact I use notepad++ to edit scripts since that at least is more sane even though it is a hassle to switch back and forth.
As far as the Lua issue, I'm sure it could be worked around. Assuming decent performance could be had in the ideal case(which I believe it could be since Lua is supposedly one of the fastest scripting languages out), then it's worth trying as I'm sure the major issues in performance could be minimized.
http://www.lua.org/manual/5.1/manual.html#2.10
It seems that Lua has manual memory management and I doubt it will be an issue with the processing function since it shouldn't be allocating anyways.
So, all that would really need to be done is for the processor to be ran in a different thread/process that the garbage collector. In that case, the only issue is accessing memory... which could be done by passing a context pointer or some ingenious tricks(e.g., use some page mappings to duplicate the memory space of the script and run the audio processor in it).
I'll try out the protoplug. While it is nice in general, having reaper sort of have that built in with access to the API would be much better. (e.g., for graphics, input, etc)
[Edit: tried protoplug and it crashed reaper when trying on of the filter examples. Some worked though]