Old 05-14-2013, 03:29 AM   #1
JasperO
Human being with feelings
 
Join Date: Nov 2009
Posts: 33
Default Midi to video plugin, is this even possible?

I've been thinking about programming a plugin that takes a midi input from the track and outputs pictures or graphics to the video window. Basically, a specific midi code or sequence triggers a picture. Either generated by the plugin or a bitmap, it doesn't really matter. Because it is sent to the video window it should be possible to render a video.

I'm wondering if this is even possible within the reaper framework. Would I have to create an VST or VSTi plugin or am I limited to audio? If this is possible, what would be the programming languages that are best?

I'm experienced with programming at a low level and up for the job if it would be possible. Learning C++ would be a challenge but I'm willing to learn. I would really appreciate it if anyone could point me in the right direction so I can start coding!
JasperO is offline   Reply With Quote
Old 05-14-2013, 04:30 AM   #2
musicbynumbers
Human being with feelings
 
musicbynumbers's Avatar
 
Join Date: Jun 2009
Location: South, UK
Posts: 14,214
Default

I can't help with coding but wanted to voice my support and say I'd love a plug in like that!

I can say that the devs would probably be happy to find a way to let us render it out in reaper via the render window which is one of the great things about this program!

Hope someone comes along soon and helps you!
musicbynumbers is offline   Reply With Quote
Old 05-15-2013, 12:47 AM   #3
Xenakios
Human being with feelings
 
Xenakios's Avatar
 
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 8,062
Default

As far as I know, the video support in Reaper is itself a Reaper extension plugin with no public API. (In contrast to the main/"core" Reaper app that does have a public API.)

The idea is interesting though, I've been wondering does there even exist any software that could do that sort of stuff easily. (Control images, videos and effects on them with rich remote control/automation envelopes. I obviously know of things like Sony Vegas or Adobe Premiere, but those are not really the kind of things I am thinking about here...) Max/MSP with the Jitter extension, perhaps...? I have already experimented with the Processing programming language, with which I was able to do MIDI controlled live still images processing, but it was very fiddly code to write and I didn't particularly enjoy writing it. Processing has video support but trying that was very unsuccesful for me...
__________________
I am no longer part of the REAPER community. Please don't contact me with any REAPER-related issues.

Last edited by Xenakios; 05-15-2013 at 12:52 AM.
Xenakios is offline   Reply With Quote
Old 05-15-2013, 05:03 AM   #4
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

It would be possible do similar things with With still pictures and the API but if video rendering is required at the end of the day, that's not possible indeed.

Quote:
Originally Posted by Xenakios View Post
The idea is interesting though, I've been wondering does there even exist any software that could do that sort of stuff easily. (Control images, videos and effects on them with rich remote control/automation envelopes.
AFAIK, only possible with hardware, not with software yet.

Some hardware can do this in real-time but *only* in the video baseband domain (i.e. SDI sources), i.e. this is what big boys use in broadcast sutdios ($$$).
In the compressed domain (MPEG-2, H.264 and now HEVC/H.265/MPEG-H), some hardware can do this too, typically to insert local ads in a national compressed live feed BUT with an unacceptable latency for the type of use we're talking about here: to do that in a frame accurate way, you have to change the GOP structure, i.e. insert an I frame at the right time. In other words you have no choice: you have to decode (quite fast) -> video FX or whatever (often slow) -> re-encode (THE cpu hogger)
^^ simplified to the extreme, lots of other bummers like VBR & video qualty preservation (RE-encoding=artifacts), some sort of PDC but for video, etc..

Software: today, a pure software solution would not cope with the bitrate of an uncompressed feed, and of course it won't beat hardware on compressed feeds - which are not real-time themselves yet.
So, as of today, there is no software soluion for *real-time*, seamless & frame accurate video edition - I am not talking about offline/render-style video edition here, of course.
I'm pretty sure we will be close to it before 2030 though In the meantime, you can control "media servers" to achieve this, like VJs.


EDIT: re-read what I wrote.. sounds a bit pompous.. To make it clear, if you don't stop me I could write a book about things like that: my day job = making such video devices. I have the stange impression I replied to one of my customers here haha! Sorry about this confusion Xen..

Last edited by Jeffos; 05-15-2013 at 05:33 AM. Reason: mailbox vs forum confusion..
Jeffos is offline   Reply With Quote
Old 05-15-2013, 07:00 AM   #5
Banned
Human being with feelings
 
Banned's Avatar
 
Join Date: Mar 2008
Location: Unwired (probably in the proximity of Amsterdam)
Posts: 4,868
Default

+1 for Resolume.

+1 for Max/MSP/Jitter.
__________________
˙lɐd 'ʎɐʍ ƃuoɹʍ ǝɥʇ ǝɔıʌǝp ʇɐɥʇ ƃuıploɥ ǝɹ,noʎ
Banned is offline   Reply With Quote
Old 05-15-2013, 11:28 AM   #6
toddhisattva
Human being with feelings
 
toddhisattva's Avatar
 
Join Date: Jun 2008
Location: Austin
Posts: 289
Default Danger ;-) Will Robinson

Already for Reaper, in JesuSonic, are Disco Lara and Slim https://stash.reaper.fm/v/5302/Disco.zip

My quest for deterministic music visualization has got me brushing up on Blender. You have the right idea using MIDI it's easy to tell pitch and timing, where the FFTs and whatnot in audio-land require real work.

So far I have a stick figure strumming six strings. Well, at least I tuned the guitar strings with each other, the low E is at 1Hz the top E is 4Hz with a lot of 8Hz harmonic.

https://www.box.com/s/uaqq5eioa9jdpgsneh28

Here you can see the winding on the wounded strings - heh heh I stretched the strings and now they're flat-wound!

https://www.box.com/s/g37euhyon94npzwa8r83

I actually have more, like a good-looking kick drum audience-facing head vibrating in a cartoony way with the "wave" modifier. I'm a long way from reading any MIDI and keying animation from it! May never get there. Don't care, this is fun. Should get a Blender music video out of it even if I never key it from MIDI.

BTW, the argument to the trig functions in Blender is the frame number interpreted as radians. So multiplying the phase by ~0.2094 gives you 1Hz at 30 frames/second. freq * 2*pi / fps

Are you thinking of something like Delay Lama http://www.audionerdz.com/

Something like a band of Delay Lamas? That's what I'm thinking, but rendered in Blender instead of Reaper, non-realtime, nice raytraced.

Or something less 3D, like MilkDrop keyed from good dependable MIDI instead of its less-than-perfect beat and pitch detection stuff? That is far more reasonable and doable in real-enough time, as a VST, in C, maybe even JesuSonic. A start would be something like Conway's Life with cells controllable by MIDI, a major chord could create a bloom of gliders, whatever.

Hah! I just now gogulated upon "Blender MIDI" I've got a lot of reading to do now ----
toddhisattva is offline   Reply With Quote
Old 05-15-2013, 05:35 PM   #7
GregHolmes
Human being with feelings
 
GregHolmes's Avatar
 
Join Date: Sep 2009
Location: Ontario, Canada
Posts: 399
Default

How about http://vvvv.org ?

Info at https://en.wikipedia.org/wiki/Vvvv
__________________
Greg Holmes | play:GregHolmes.com | work:GHServices.com
GregHolmes is offline   Reply With Quote
Old 05-16-2013, 05:01 AM   #8
JasperO
Human being with feelings
 
Join Date: Nov 2009
Posts: 33
Default

Thanks guys for all the feedback! I really have to look at all the possible ways I might get this done.

What I use right now are video files that were converted from single images. (png -> mp4) I'm using them in my projects. So at certain positions in time you see a particular image and I use a lot of loops.

There are two reasons why I would like to trigger them from midi: 1. reaper uses a lot of memory when handling videos (mp4). I have 3GB and run Win7 but I'm considering buying more, just to run reaper. 2. Loops do not work properly and I do not expect it to be fixed as it has been known for several years. When flipping back reaper keeps the last frame for half a second or so.

So I just found out that I could use the PNGs directly in reaper (instead of mp4s) but the image preview in the track cannot be turned off (AFAIK). So you see a preview of the picture when you drag it on a track. You no longer see the track name anymore. Like the mp4s, the loops do not work properly. Videos don't have this preview but will eat GBs of memory.

Although I actually do not necessarily need to trigger by midi it would give me a lot of extra benefits. At least loops will work and when I create a program I have full control over the outcome. But unfortunately, as the video API is not readily accessible it seems impossible.

So basically I would like to have images sent to the video window on fixed positions that loop properly and without the preview on the track that do not need GBs of memory. The blocks on the track are important pieces of information and I would just like to keep them a single color with a text on them. By using midi I could have midi information on certain fixed positions in time and trigger a plugin instead of using the images or still videos directly in the project.

I hope you could give me some advice about how I can better integrate these images (turn the preview off and getting the loops to work). It is a bit off topic to my questions but that's actually where I need the midi plugin for.
JasperO is offline   Reply With Quote
Old 05-19-2013, 03:00 AM   #9
Jeffos
Mortal
 
Jeffos's Avatar
 
Join Date: Dec 2008
Location: France
Posts: 1,969
Default

Quote:
Originally Posted by JasperO View Post
I hope you could give me some advice about how I can better integrate these images (turn the preview off and getting the loops to work). It is a bit off topic to my questions but that's actually where I need the midi plugin for.
Well, on the contrary I think you are the only guy on-topic here haha!
I'm sorry you didn't get much help but personnaly I'm just not sure I follow... The MIDI triggering thing first let me thought you were after live interaction. Then, it seems you know you can drag-drop still prictures in the arrange and render to video, but I don't understand the "turn the preview off" part (images are actually hidden from video window/render on item ends).

I can just say there's nothing related to video in the API.
The MP4 looping that keeps "the last frame for half a second or so" is "normal" : it's exactly the kind of thing I was talking about in my previous post (REAPER is most probably wating for for the next GOP on loop).
Quote:
Originally Posted by Omfoo View Post
there certainly are full-blown real-time applications that can do it.
I'm not sure if you're posting this following my remarks..
Just to clarify, like I said, yes, there are VJ apps for playout!

(Can't resist to post some links too...)
But for live videos feeds & processing.. Well, you can do things too, but in degraded/low resolutions ATM - It can be surprisngly cool though!
Ex: one of my best friend has designed this: http://vimeo.com/52924728. As of now, they use Madrix to control this (or a set of these).
A hacked version of Cockos' LICEcap is feeding the beast too
Jeffos is offline   Reply With Quote
Old 05-19-2013, 04:03 AM   #10
MichaJo
Human being with feelings
 
MichaJo's Avatar
 
Join Date: Oct 2008
Location: Germany
Posts: 1,772
Default

Maybe i missed the target, but could Pure Data be of help?
http://puredata.info/

About 7 years ago i've made some tests to get Blender 3D connected with MIDI/OSC/Python etc. to do stuff like animusic does(did?).
Example of animusic: https://www.youtube.com/watch?v=lQso49UOW84

PureData has been one of the components i've tested then.
I'm totally out of this topic now, but it seems to me PureData could be worth mentioning?

Michael
MichaJo is offline   Reply With Quote
Old 05-19-2013, 04:24 AM   #11
bennetng
Human being with feelings
 
Join Date: Jun 2010
Posts: 264
Default

MIDITrail - A MIDI player which provides 3D visualization
http://youtu.be/w15FGxp96-Y
bennetng is offline   Reply With Quote
Old 05-19-2013, 04:45 AM   #12
G-Sun
Human being with feelings
 
G-Sun's Avatar
 
Join Date: May 2010
Location: Norway
Posts: 7,318
Default

Check out this:
http://www.olehamre.no/Default.aspx?pageid=772
(maybe open in google translate)

Use your keyboard to play
__________________
Reaper x64, win 11
Composer, text-writer, producer
Bandcamp
G-Sun is offline   Reply With Quote
Old 05-20-2013, 04:58 AM   #13
JasperO
Human being with feelings
 
Join Date: Nov 2009
Posts: 33
Default

Quote:
Originally Posted by Jeffos View Post
I'm sorry you didn't get much help but personnaly I'm just not sure I follow... The MIDI triggering thing first let me thought you were after live interaction. Then, it seems you know you can drag-drop still prictures in the arrange and render to video, but I don't understand the "turn the preview off" part (images are actually hidden from video window/render on item ends).
When you drag an image onto a track, you will see a block on the track with a thumbnail of that image. That thumbnail or preview is what I would like to turn off. The thumbnails are repeated from the left to the right and the track view becomes very messy.

A bigger problem is that when you drag many pictures on a track, reaper does not display half of them. It is very random and this looks like a bug. I tried to render a two minute video of about 20 images to a very low resolution movie and it took about 3 hours (!!). The rendered movie missed frames and had a lot of weird blocks. Replacing the images on the track with mp4s that show the same static image (basically the same result) takes 30 seconds to render and the resulting movie is perfect. But this uses GBs of memory.

So I do not really need to display images that are midi triggered, but it allow me to bypass the video engine of reaper. Also it would give me a lot more flexibility that I do not have right now.

Quote:
Originally Posted by MichaJo View Post
Maybe i missed the target, but could Pure Data be of help?
http://puredata.info/

About 7 years ago i've made some tests to get Blender 3D connected with MIDI/OSC/Python etc. to do stuff like animusic does(did?).
Example of animusic: https://www.youtube.com/watch?v=lQso49UOW84

PureData has been one of the components i've tested then.
I'm totally out of this topic now, but it seems to me PureData could be worth mentioning?
Michael
Cool! that looks like it could work ;-)
JasperO is offline   Reply With Quote
Old 05-22-2013, 12:23 PM   #14
toddhisattva
Human being with feelings
 
toddhisattva's Avatar
 
Join Date: Jun 2008
Location: Austin
Posts: 289
Default Bleaper Video

Twenty-One Ball Salute!

Animated in Blender with MIDI and audio from Reaper. It plays well in VLC but on my iPad the A/V sync isn't. Oopsies.

Top row is Bezier anticipatory response (so player's motion starts before the note and finishes when the note starts), the middle is Constant response on at note-on and off at note-off no anticipation, and the bottom is a linear anim curve, anticipatory.

Here's the Python script that you run in Blender that reads a MIDI file and makes velocity-sensed "levers" (empty axes) that you use their Z value in Blender animation drivers (a very cool animation feature not at all realated to the scary assembly language interrupt land drivers for hardware!).

https://www.box.com/s/df6au5uzmexp6fbgo8a3

There are variables you'll want to change at the top of the file, these are commented. You'll need to change the filename.

The code is un-commented and somewhat silly, just one week of Blender scripting experience here. Apologies.

You need http://www.pjb.com.au/midi/free/MIDI.py

Put it in .../Blender/2.66/python/lib

Just put it in all of your Python 3 installs! Then you'll have it in Reaper. It's great!

I learned what Actions are and the NLA Editor is last night. Have messed with them from Blender's Python window. Plans.
toddhisattva is offline   Reply With Quote
Old 01-11-2016, 09:58 AM   #15
vanhaze
Human being with feelings
 
vanhaze's Avatar
 
Join Date: Jul 2012
Location: Netherlands
Posts: 5,247
Default

Love to bring this thread up again ..

Would be awesome to trigger parts of a video track like a sampler plugin for audio in Reaper :

You just make selections of a video, dump those to an almighty wished for "video sampler" plugin in Reaper.

You then make a midi item in which different midi notes trigger different video clips that were dumped
into the almight video sampler plugin.

Should i just keep on dreamin" ??!! :0)
vanhaze is offline   Reply With Quote
Old 01-15-2016, 10:48 AM   #16
helgoboss
Human being with feelings
 
helgoboss's Avatar
 
Join Date: Aug 2010
Location: Germany
Posts: 2,185
Default

Quote:
Originally Posted by vanhaze View Post
Love to bring this thread up again ..

Would be awesome to trigger parts of a video track like a sampler plugin for audio in Reaper :

You just make selections of a video, dump those to an almighty wished for "video sampler" plugin in Reaper.

You then make a midi item in which different midi notes trigger different video clips that were dumped
into the almight video sampler plugin.

Should i just keep on dreamin" ??!! :0)
That sounds like a job for Playtime. It's a sort of item sampler. It can also trigger image or video items. Upcoming version 1.11.0 makes it easier to play sections of a video clip. For example, it will preserve the length and cut of the video on the timeline when filling the video item in a Playtime slot (that was not possible before). A prerelease can be downloaded in this post.

Playtime's strength is tempo-syncing, so by default it starts the video on the next measure. Recommended settings for firing the video clip right when you press the key:

- MIDI toggle mode: Off
- Play start time: Immediately
- Exclusive mode: Off
helgoboss is online now   Reply With Quote
Old 01-15-2016, 01:29 PM   #17
vanhaze
Human being with feelings
 
vanhaze's Avatar
 
Join Date: Jul 2012
Location: Netherlands
Posts: 5,247
Default

OMG, Insane !!!!
vanhaze is offline   Reply With Quote
Old 02-01-2016, 09:54 AM   #18
vanhaze
Human being with feelings
 
vanhaze's Avatar
 
Join Date: Jul 2012
Location: Netherlands
Posts: 5,247
Default

Just tried and i am not getting it ..

- Video Track with video clip.
- Track with Playtime on it.
- I split some short video clips out of the main video clip.
- I load these short clips in Playtime slots.
My goal is to just have the main video clip playing and then trigger the filled playtime slots with my maschine studio controller, quantized on grid.
These triggered slots (short videoclips) must be recorded and will "replace" the main video track.

So when played back, you see main videoclip, which then get's "overrided" by the triggered short video clips. When there is no triggered videoclip on timeline, the main video track must be visible. (continue to play)

Sorry if i explain things clumsy ..

If this is possible at all, then i would love to see a tutorial on how to exactly setup things in Reaper for this.
vanhaze is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 02:06 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.