Well, the actual idea is not too hard -
here's a proof-of-concept demo that I just knocked together. That doesn't cache the files at all (so it will get multiple handles for the same file if multiple notes are using the sample), and it uses simple linear interpolation, but it still works.
I was more wondering what kind of performance issues or whatever I might run into, or if there are technical limitations to this kind of approach that are going to scupper me months later.
For example, many sample-based synths that stream from disk will buffer ahead, but that's something you can only do if you have threads, so not being able to do that might have latency implications. Does using file_open() with a number (instead of a string) keep some kind of reference to the file in the file system, or is resolving the filename every time the file is opened?