Integrate score editor as an extension

[pcartwright]

I originally put this in the general forum (http://forum.cockos.com/showthread.php?t=47255), but I realized that the developer's forum may be more appropriate.

I’ve been using REAPER for about a year now (and I love it!), but I am in desperate need of a score editor. My work around to the lack of a native music notation editor has been to use MapleMIDI (internal MIDI connections) to connect Finale to REAPER. This process is at best inefficient and tends to eat up a lot of computer power and time.

I know this score editor request has been beaten to death, but has anyone considered applying a score editor as an extension (much like the SWS extensions)? I think it would be possible to set up a system that could point to native MIDI information, offered editing capabilities, and a decent amount of customization for those who want/need an internal score editor. And for those who don’t want a score editor, don’t add the extension.

Grant it, I’m not a computer programmer (and I’ve never tried to write extensions for REAPER), so maybe this is a bigger undertaking than I realize, but I’d be willing to learn the necessary programming and assist anyone who would be willing to help me with this concept.

I’ve got several ideas on how to make use of REAPER’s designs and interfaces to make this a very versatile utility, so if anyone’s interested, please let me know.

[sfzgeek]

A score editor would be a massive undertaking, by my reckoning.

[pcartwright]

That's what I was afraid of. Do you know if there are any good starting points to work from? I guess I'm thinking integrating or porting an open source editor as an extension would be the easiest solution (if a solution exists).

[Xenakios]

Originally Posted by pcartwright

I guess I'm thinking integrating or porting an open source editor as an extension would be the easiest solution (if a solution exists).

Existing Open Source code is likely to be licensed under the GPL license, which is very problematic. Apparently the GPL license does not allow to do plugins to programs like Reaper. You could of course do for your private use anything you want, but if your plan is to distribute (even for free of charge), you don't want to be violating any licenses.

[MikeLacey]

"Apparently the GPL license does not allow to do plugins to programs like Reaper"

We could always ask the author(s) if there's one which looks really suitable; given permission you can do anything.

[Xenakios]

Originally Posted by MikeLacey

"Apparently the GPL license does not allow to do plugins to programs like Reaper"

We could always ask the author(s) if there's one which looks really suitable; given permission you can do anything.

Yes of course, but if even one contributor declines or can't be reached anymore (deceased, not interested anymore etc), it's pretty much a dead end.

By the way, the interpretation of GPL that disallows doing plugins to proprietary programs like Reaper, is based on a reply I received from the main author of a certain GPL'd library. That interpretation may not be entirely correct. (The author in question also sells commercial licenses to his library, so I guess it was in his interests to give me a discouraging reply.) The best way to know would be to ask a lawyer specialized in software licenses...

[MikeLacey]

Hmmm... And choosing a suitable one is non-trivial, most of them seem to be full of bells & whistles. We'd (ideally) want a very small, efficient and focussed score editor.

You have one in mind Mr/Ms Cartwright?

Mike

[Hartley Mays]

I'm another one of the many Sonar refugees, and share the feeling that a midi score editor is the biggest comparative shortcoming of Reaper for now.

I've done some work and communicated with the authors of two open source Java projects which have score editor, Frinika and Improvisor. Regardless of intepretations of the GPL, I think the authors would probably be agreeable to potential use with Reaper, and I could check with them about it if the functionality of one of these editors might suffice. Both are hosted on Sourceforge.

Another Sourceforge project, JVSTWrapper, could be used to bridge the Reaper and Java environments. It provides the JNI interface to develop VST's in Java; I've worked some with this project also.

Here are the links to the three projects:

http://frinika.appspot.com/

http://improvisor.sourceforge.net/ (There's a Yahoo group for this project also which is more active than the sourceforge site.)

http://jvstwrapper.sourceforge.net/

Later,
Hartley

[pcartwright]

Originally Posted by MikeLacey

You have one in mind Mr/Ms Cartwright?

Mike

So far, I've looked at Canorus and MuseScore. I don't think either of those progams have exactly the functionality needed... Canorus seems to be too flexible and requires a bit of work just to add basic items (bar lines, time signatures, etc) while MuseScore may have too many bells and whistles.

Thoughts?

Oh, and Mr. Cartwright would be fine if you'd like. lol

[MikeLacey]

Mr Cartwright it is then.

Can you have a look at Frinika and Improvisor from Hartley's post? I will as well but I suspect you've done a bit more thinking about it than I have as yet.

Hartley - is a VST the way to go? There is also an option to define an external editor for media items. A VST though, would have the advantage of not interfering with the standard REAPER way of working...

[Hartley Mays]

I would assume that the external editor would be a lot less work; probably it could be done first to make the editor (whichever one chosen) available and see how it works out. I've just started to get familiar with the Reaper extensions to the VST SDK, and we would need to figure out how much additional functionality would be available by using the VST and extensions approach. I've done a decent amount of development work in Java, but have limited C++ experience, lookig forward to it as a good learning opportunity.

It would be good to know whether the Reaper developers already have work underway on the editor, before we devote a lot of time to it. I've seen a lot of discussion from Reaper users about it, but nothing from them either way.

I'll do some more research on it including looking at the other editors mentioned above later today.

Hartley

[MikeLacey]

Good question.

From what I recall of the discussion I followed Justin et al weren't that keen, we should ask, I'll ping schwa.

[MikeLacey]

I doubt it would remain on any to-do list if there was a good Score Editor available as an external editor/VST/extension. No "road map" forthcoming though.

[pcartwright]

Originally Posted by MikeLacey

Mr Cartwright it is then.

Can you have a look at Frinika and Improvisor from Hartley's post? I will as well but I suspect you've done a bit more thinking about it than I have as yet.

Hartley - is a VST the way to go? There is also an option to define an external editor for media items. A VST though, would have the advantage of not interfering with the standard REAPER way of working...

I hadn't heard of Frinika until Hartley mentioned it; I'll be sure to give it a try and see how it works. I've tried improvisor fairly recently and wasn't really impressed, but I'll try it again in order to get a better grasp on the benefits of the program... Also, Frinika may be the easiest to integrate considering the score editor is already programmed into a DAW platform.

Sorry to speak out of turn, but I do want to address the VST/extension question.

I do think that creating an extension makes more sense than creating a VST plugin. Again, I'm no computer engineer, but purely from a user standpoint it seems like we would be limited to viewing and editing one track at a time with a VST. The beauty of an extension is that you could potentially view and edit multiple tracks in a single window. If the same can be done in a VST format, then let's consider it, but otherwise I would press for an extension version.

[MikeLacey]

"Sorry to speak out of turn"
Don't be daft

Having said that though...

"The beauty of an extension is that you could potentially view and edit multiple tracks in a single window."

It might also make it a bit more complex

[pcartwright]

Originally Posted by MikeLacey

"Sorry to speak out of turn"
Don't be daft

Having said that though...

"The beauty of an extension is that you could potentially view and edit multiple tracks in a single window."

It might also make it a bit more complex

No doubt, but I think the headaches in the integration would we worth the flexibility (and lack of headaches) during production use.

[MikeLacey]

Point taken.

One step at a time I suggest, something suitable and permission to use it is the first one.

[pcartwright]

Originally Posted by MikeLacey

One step at a time I suggest, something suitable and permission to use it is the first one.

Agreed.

Mike and/or Hartley, do you have any experience obtaining permissions like this? Again, I don't have much experience, so I don't know what you would need (i.e. written authorization letter or something similar).

[Hartley Mays]

Here are some thoughts and questions on the points raised since my last post.

I have some commercial experience with open source license issues from my "day job" at a Cincinnati-based software company. I think that email permission from the owner of the copyright would suffice and can check further on it. We also need to be sure exactly which license is involved; one of the problems with the open source trend is the proliferation of licenses. For instance, there's the LGPL which is a variation of the GPL that permits commercial use.

I don't think any license issues would be involved with the external editor approach.

As I mentioned earlier, I haven't worked yet with the Repaer VST SDK extensions, but have it downloaded along with Visual C++ Community Edition 2008 ready to go. Is the use of that extension kit what's meant by doing it as an extension rather than a VST? I think there are probably a lot of reasons that a VST approach would be pretty limiting; I don't think the editor could easily request data because the host would be in control of the transmission.

I'll look now at the four editors in more detail; there are other open source projects I'm aware of with editors, but I haven't worked on any of them nor contacted the authors. I know that the Frinika authors are very cooperative on any questions and potential use of their product.

Later,
Hartley

[MikeLacey]

"Is the use of that extension kit what's meant by doing it as an extension rather than a VST?"

That's right yes - and I was wondering that about the VST approach as well. The host gives (as I understand it) data to the VST as the DAW plays - so a Score Editor would only be able to display the complete score once the piece had been played all the way through.

REAPER's got some good API functions to Get and Set Track and Item data.

You mentioned that you have the Visual C++ Community Edition 2008 - I have VC++ Express...The requirement for developing REAPER extensions is "MSVC++", it says, that's the only information I have though.

SWS & Xen, can you advise which C++ environment we should be looking at?

[MikeLacey]

Frinika looks slick, not so keen on Impro-Visor (both on first look)

[EvilDragon]

So let me get this straight... you guys are sharpening your teeth on making a new Notation/Tracker view in MIDI editor in Reaper, as an extension?

[Hartley Mays]

Mike,

As far as I know, we're referring to the same MS C++ compiler, and it works for all MS C++ code; the community edition IDE just doesn't allow direct development of resource files for GUI's. I think there's been some other posts in this forum about the requirement and benefits of using the native MS compiler instead of the alternatives.

Were you referring to the overall projects or specifically to the score editors when comparing Frinika and Improvisor?

Interestingly, one of the two main Frinika developers is a professor in southern England, while Improvisor is hosted at McGill University in California, close to Reaper's home. I think that a lot of the development work on Improvisor is done by students on their way through the music program there, so it may be more difficult to get good advice. I haven't tried to work with any of that code yet, but have done a significant amount of work on the Frinika code (including actually the score editor) to support the sysex messages from Rodgers organs. In general, the code is well organized and straightforward, but the score editor, in tribute to its domain, is (necessarily?) hard to follow.

Later,
Hartley

[GregHolmes]

VST would not be the way to go.

One reason is that it would only see one track at a time. A score editor would be far better with a view of all tracks, each with its own clef and staff layout (e.g. grand staff, percussion, etc).

Another reason is that the VST spec is not flexible enough for this purpose. Ask Ralph at Rayzoon/Jamstix about that... :-)

Of course, a VST could have special Reaper hooks, but with the amount of work for that (both in the score editor and in Reaper itself), you might as well build a Reaper extension with cooperation from Cockos.

I used Jim Miller's Personal Composer in the 80s, and it was a fantastic tool, with an excellent score view (which I needed in order to copyright the compositions). There isn't much today that matches the immediacy and utility of that program. For me, it's the benchmark.

[Hartley Mays]

Well, we're hopefully sharpening interest and enthusiasm anyway.

The good ideas are the hard part, the rest is implementation details....

[MikeLacey]

This is a bit like wading through treacle, but without the sense of achievement you get when you're up to your neck in ...

There are a lot of sweet & sticky Score Editors out there and after wading through a few Frinika *still* looks slick.

It's in Java though

I know, Java's wonderful - but integrating an existing complex and very functional something into REAPER is going to be pain enough without adding the language barrier.

Hartley, I don't know the first thing about Java but you have done work on Frinika. Given the appropriate permissions is it feasible to integrate a Java thing into a C++ application? I don't know if calling the REAPER API would be possible, I'd guess not really so I've been assuming that we'd need some ReaScript glue in Perl (or that *other* language) to get data from, call a Java app and to pass data back to REAPER.

Am I thinking along the right lines?

[whatsup]

Originally Posted by MikeLacey

This is a bit like wading through treacle, but without the sense of achievement you get when you're up to your neck in ...

There are a lot of sweet & sticky Score Editors out there and after wading through a few Frinika *still* looks slick.

It's in Java though

I know, Java's wonderful - but integrating an existing complex and very functional something into REAPER is going to be pain enough without adding the language barrier.

Hartley, I don't know the first thing about Java but you have done work on Frinika. Given the appropriate permissions is it feasible to integrate a Java thing into a C++ application? I don't know if calling the REAPER API would be possible, I'd guess not really so I've been assuming that we'd need some ReaScript glue in Perl (or that *other* language) to get data from, call a Java app and to pass data back to REAPER.

Am I thinking along the right lines?

I think I saw some C freeware scores on sourceforge.
to begin with Java or a script, i think it's a wrong start.

[Hartley Mays]

Mike,

In my initial post on this thread, I referenced the JVSTWRAPPER project, which provides the necessary interface between C++ and Java for the VST SDK. It's not easy; each call must pass through the so-called "JNI" layer, Java Native Interface. The JVSTWRAPPER code would need to be extended with whatever additional calls are in the Reaper Extensions, assuming that we don't do it as a regular vst, per the above discussion.

I've begun working on modifications to the 6.0 Frinika code so that it can be called directly as an external editor; right now it doesn't accept any runtime arguments. That would make it easier for people to try out, and might provide a lot of what would be needed without fooling with the direct interface at all. We could add some additional code directly into Frinika to save the output directly where Reaper expects it, perhaps take advantage of the straightforward structure of the Reaper project files to extract multiple midi tracks, etc....(sort of speculating as I write here if you hadn't noticed...)

I'll email the Frinika authors tomorrow.

All that said, I agree with Whatsup that we should also keep looking at C++ alternatives.

Later,
Hartley

[Geoff Waddington]

I've been watching this thread from a general interest perspective, as well as from a programmer perspective.

I've succesfully built a proof of concept app in C# using .NET that allows bidirectional communication with Reaper using the C++ SDK.

So, speaking from experience this is not a trivial undertaking, and has a large amount of "pass through" code much like the JNI approach mentioned here.

I'm jumping in at this point to ask a couple of questions that I don't see clear answers to in this thread so far -- maybe I'm just too dense to see those answers

The external editor approach is attractive from an isolation / modularization perspective, while heavier integration leads more toward a realtime based approach, albeit at a greater cost.

So some questions:

What are the requirements that would make users happy?

Is it sufficient to just be able to print out a score derived from a number of MIDI tracks -- this info is likely reasonably available at fairly low development cost (time / effort, etc.) by simply using something like Perl to parse the project file and feed the resulting stream to the editor, or even more cheaply by simply generating macros / scripts to export project MIDI files.

Is the requirement "heavier" with a need for realtime interaction, e.g. editing notes in the existing piano roll editor is immediately reflected in the score and vice-versa?

Is the approach more "Let's just experiment to see what's possible" before a decision is made on what form the project should take.

I think I've seen hints towards all three of the above options "between the lines" of the various posters so far.

So to reiterate -- What would a succesful integrated scoring solution for Reaper look like?

[EvilDragon]

Originally Posted by Geoff Waddington

So to reiterate -- What would a succesful integrated scoring solution for Reaper look like?

Score view in MIDI editor. That's right, fourth view

[pcartwright]

Originally Posted by Geoff Waddington

Is the requirement "heavier" with a need for realtime interaction, e.g. editing notes in the existing piano roll editor is immediately reflected in the score and vice-versa?

IMHO I think realtime interaction is an important piece (if not THE most important piece) of this effort for two basic reasons.

First, user time efficiency.

There are ways you can send MIDI tracks from Finale (or other external score editors) to REAPER via virtual MIDI devices. So, I don't know if another external editor would be that much better unless it were much more efficient (in terms of computer resources). However, even with all the computer power in the world, composing with Finale and REAPER at the same time requires editing MIDI components in the external editor and REAPER as the composition is being formed or edited.

Some score editors are better for MIDI controller data than others, but I can't think of a single score editor that comes close to the MIDI functionality (routing, plugins, etc) of REAPER. Even if you were able to compose entirely in a score editor, you would still have to tweak some settings in REAPER if you wanted to make changes to your composition.

An internal score editor with realtime interaction would allow composers to make changes in only one program during the composition phase. From my experience with Sonar and now with REAPER, it's much easier to be able to edit in one system (i.e. Sonar with score view) then in two systems (i.e. Finale and REAPER). Simply put, running two programs takes much more time than running one program.

Second and probably more importantly is system efficiency.

I have a pretty nice system for composing; however, I have noticeable latency issues when Finale and REAPER run simultaneously. I'm no computer expert, but I think that I could save RAM and CPU power if the score functionality were in the same program as the instrument and effects plugins instead of trying to connect the two programs through virtual MIDI devices.

A few other thoughts:

I'll save my thoughts on what functions should be included until we have a better understanding of what we can and can't do. I'm not a computer programmer, so my ability to help on the technical side is very limited. I'm just explaining what I need in order to be more efficient with REAPER.

I do not need an editor that looks "pretty." Sonar's score editor was useful, functional, but it was not easy on the eyes (especially when dealing with complex rhythms and ledger lines. I think programs like Finale, Sibelius, etc should be used to make any final printable version of a piece. These programs are built to look professional (not necessarily sound professional).

As far as printing sheet music, it would be nice to be able to print some sort of music notation from REAPER, but it wouldn't be necessary. I certainly don't think we should burn resources trying to add aesthetic appearance in lieu of other useful functionality.

*******

If anyone disagrees with my interpretation of the potential editor's form and direction, by all means give your opinion. I'm just outlining something that I think would add a great deal of functionality to those of us who rely more on music notation than piano rolls.

[pcartwright]

Sorry that last post was so long; I just wanted to make sure everyone understood my perspective.

[GregHolmes]

Personally, I'm not a fan of piano roll views or event list (tracker) views. They are like guitar tab - they show the motions, not the music.

I like real music notation, but there will always be a disconnect between the rigidity of music notation and the flexibility of non-quantized human performances. But that's as it should be. A notation view should VISUALLY quantize, while leaving the underlying events un-quantized. It's useless if the notation view is obscured by 128th note triplets and tie lines all over the place, because it is trying to show the messy timing of the underlying events.

I believe that a notation view will have to be completely integrated into Reaper, so it has full access to the data. For example, moving a group of notes in the notation view must also move all the related non-note events in the event list. Another example: if you have a bunch of notes that were recorded with loose timing, then use the notation view to copy and paste them, they should retain the loose timing when they are pasted (even if they look like a bunch of evenly played 8th notes). Of course, new notes that are entered in notation view will be "right on the grid", so there should be a way to nudge them off it, without changing the way they look.

FYI, I am a programmer (but not the guy to work on this project), and I have been using music notation software since the previously mentioned Personal Composer ran on DOS.

[carbon]

Would Denemo qualify as an alternative:
http://www.denemo.org/index.php/Main_Page

Thought that I'd mention it here, just in case....

[zappsunzorn]

How about expanding the functionality of the export to external midi editor function that we now have. Perhaps making a software specific format adapter with adjustable parameters inside Reaper. Could this be made to work seamlessly enough, and how hard would it be to get to use the formats for the other software (literally and legally)?

Edit: this would not work without some kind of music notation/midi hybrid file format, a format that held both the performance and notation of that performance.

[MikeLacey]

From reading the user manual http://www.denemo.org/doc/denemo-manual.html it seems that Denemo aims to be a useful front-end to Lilly Pond. So it's nice but not, as far as I can see, a good alternative unless I've completely misread the docs.

Geoff Waddington makes some good points and a few people have tried to be more specific since his post.

Here's my one sentence definition, I think that if you can express your requirement in a single sentence you have a chance of filling it - and that your chances of success disappear rapidly as you have to add sentences. I invite everyone to do the same *grin*, one sentence.

My ideal is a starting point, a minimal score editor written in C++ that can be easily modified to read and write REAPER's MIDI data.

[LCipher]

Originally Posted by EvilDragon

Score view in MIDI editor. That's right, fourth view

+1 to show support. Even if its not true "notes" in terms of duration but just like a piano roll on a staff "grid".




edit to add link to site: http://buchla.com/historical/b400/

[MikeLacey]

@LCipher -- that is very cool...

[pcartwright]

Originally Posted by LCipher

Even if its not true "notes" in terms of duration but just like a piano roll on a staff "grid".

I think if anyone is going to make the effort to add score notation, they/we should at least go as far as adding true rhythm notation.

[pcartwright]

Originally Posted by LCipher

+1 to show support.

I do like the simple layout on the whole.