MPL Dump Retrospective Record log (no JSFX required)

[_Stevie_]

All good man! I just wanted to point it out. Looking forward to it

[mschnell]

Originally Posted by lb0

Eugen's original JSFX did it differently - and calculated the actual playback position according to the block start and offset - which would IMO be far more accurate.

If running in "playback" mode this might - or might not - help. If running in a kind of "live" (aka stop) mode, it will not.

see -> https://forum.cockos.com/showpost.ph...3&postcount=90
- doing anything related to Midi in @sample does not help at all.
- in @block the "offset" is relative to the block that is currently worked on in the effect chain. this might be another block than currently being worked on in another OS thread (another track or the GUI).

Supposedly, the playback position denotes the correct block and offset values the Midi information needs to be related to (no idea how this can be done).

-Michael

[lb0]

Originally Posted by mschnell

If running in "playback" mode this might - or might not - help. If running in a kind of "live" (aka stop) mode, it will not.

see -> https://forum.cockos.com/showpost.ph...3&postcount=90
- doing anything related to Midi in @sample does not help at all.
- in @block the "offset" is relative to the block that is currently worked on in the effect chain. this might be another block than currently being worked on in another OS thread (another track or the GUI).

Supposedly, the playback position denotes the correct block and offset values the Midi information needs to be related to (no idea how this can be done).

-Michael

Hey Michael,

Thanks for the info - very interesting read.

So with respect to incoming 'live' MIDI - this is simply inserted in the queue as soon as it arrives at the fx input? No buffering of live midi (which would make it late anyway)?

If so - then my understanding was slightly off - and perhaps mpl's method is just fine or as reasonably accurate as can be expected? (at least when transport is stopped)

Thanks for your insight.

EDIT:

Maybe I misunderstood your original comment?

There is no 'live' communication via gmem between tracks. the data is simply stored in global memory so it is ready to be read when capturing is complete via a separate lua script.

But what you're getting at is that live midi coming into one track will have different timing as the same live midi coming into another track (with different plugins)? Therefore what is captured by RetroRec is never going to be exactly the same as what you hear live?

Sorry for all the questions - just trying to get the best understanding possible...

[lb0]

I've tested this script alongside Eugen's and also captured the same midi directly onto a track.

From what I can tell - Eugen's is as near perfect compared to the direct midi. mpl's is a little looser. Not terrible - but noticeably.

But - mpl's can capture when stopped - and I'm not yet sure whether Eugen's method can be performed while stopped (EDIT: it can ).

[mschnell]

Originally Posted by lb0

There is no 'live' communication via gmem between tracks. the data is simply stored in global memory so it is ready to be read when capturing is complete via a separate lua script.

gemm is not "save" or "correctly timed" between multiple OS threads. Each track features it's own OS thread and the "guts" of Reaper (including all the GUI - including the JSFXes' @gfx sections - and the ReaScripts) has it's own OS Thread as well. As obviously a LUA script runs in the GUI thread of Reaper's, the said timing issue can hit (unless appropriate precautions are taken).

Maybe with this application (with no "Input" FX plugins enabled), the input threads usually are ahead of the GUI thread, this might "just work", but with a multiCore hardware the timing between OS threads might be very different from what we expect.

-Michael

[lb0]

Originally Posted by mschnell

gemm is not "save" or "correctly timed" between multiple OS threads. Each track features it's own OS thread and the "guts" of Reaper (including all the GUI - including the JSFXes' @gfx sections - and the ReaScripts) has it's own OS Thread as well. As obviously a LUA script runs in the GUI thread of Reaper's, the said timing issue can hit (unless appropriate precautions are taken).

Maybe with this application (with no "Input" FX plugins enabled), the input threads usually are ahead of the GUI thread, this might "just work", but with a multiCore hardware the timing between OS threads might be very different from what we expect.

-Michael

Yes - but all the JSFX does is buffer what's coming in - storing the timing and other message details of the input midi. By using the block details and offset information - this is able to capture the timing of the incoming midi perfectly.

In my tests - Eugen's script captures exactly what MIDI is input (compared to hitting record on an empty track - the buffered data is identical). I've also adapted it so it will capture while the transport is stopped (by keeping a counter which gets incremented as each block is run).

This is all that is required for this retrospective record functionality to work.

This version of the script - does not take into account midi message offset - and uses a simplified routine to work out the timing of the incoming MIDI - which does mean that the captured data does not match that of the midi data recorded directly in a midi item - although it is very close.

There must be buffering of incoming 'live' midi data - or else the midircv function would not provide an offset value > 0 for this data (which it does). So I guess this is all part of the default system latency for midi data.

Anyway - I'm happy with the solution I have got - using an adaptation of Eugen's original script. I'll provide this to mpl should he be interested and wish to adapt it for gmem use (something I'm less fussed about - for me the ability to accurately buffer the midi data when stopped was the main purpose).

[Stevie]

Originally Posted by EvilDragon

Great work (as always) mpl, but man, wouldn't it be so much easier if this was just built into Reaper, no fussing with JSFX or anything?

as huge amount of different scripts here and there ><

[mpl]

I recently added a @block offset compensation subtracted from time_precise(), so MIDI events now should be more precise.

[deeb]

Originally Posted by EvilDragon

Great work (as always) mpl, but man, wouldn't it be so much easier if this was just built into Reaper, no fussing with JSFX or anything?

this would be so great as native! no preparation of track, always there for rescue : )

It happens to me so manny times no be able to reproduce and record what i just did ! both midi notes and automation.

Would be wonderful being able to do this:
- Load last cycle touched param
- Load last cycle midi (notes or automation)
- Merge last cycle touched param
- Merge last cycle midi (notes or automation)

Options:
-buffer size cycles: [1]
-buffer seconds ( if no cycle defined): [200]
- always try to paste all buffered midi data and adjust by offset: [true/false]

And for my real use case, i think would be enough that the buffer just keeps track of: last selected track and focused instrument or FX. But things could go crazy and track everything. Anyway! dreaming : )

[zookthespook]

Hello MPL,
Hope you are good !

I cannot seem to make the script work for some reason,

1. Installed the four scripts

2. Made a new track and ran the script " Script: mpl_Prepare selected track for
RetrospectiveRecord tracker.lua "

3. Assigned a hotkey to Script: mpl_Dump RetrospectiveRecord tracker log to selected
track.lua


4. Created a new track with a VSti. Played a few notes and hit the hotkey,

So after hitting the hotkey, the script drew and EMpty midi item box , but no midi events were dumped in it .


Is there something wrong i am doing here ?


i am on

Reaper 5.979 {x64}-Reapack 1.2.2 - sws 2.1


regards
zook

[mpl]

Originally Posted by zookthespook

Hello MPL,
Hope you are good !

I cannot seem to make the script work for some reason,

1. Installed the four scripts

2. Made a new track and ran the script " Script: mpl_Prepare selected track for
RetrospectiveRecord tracker.lua "

3. Assigned a hotkey to Script: mpl_Dump RetrospectiveRecord tracker log to selected
track.lua


4. Created a new track with a VSti. Played a few notes and hit the hotkey,

So after hitting the hotkey, the script drew and EMpty midi item box , but no midi events were dumped in it .


Is there something wrong i am doing here ?


i am on

Reaper 5.979 {x64}-Reapack 1.2.2 - sws 2.1


regards
zook

Hmm try to inspect MIDI, maybe it is somehow shifted (try see in Midi editor with event list mode)

[zookthespook]

Originally Posted by mpl

Hmm try to inspect MIDI, maybe it is somehow shifted (try see in Midi editor with event list mode)

Sure will try that first thing in the morning and report back !
thank you

regards
zook

[Supremo]

Hi Mikhail!

I'm encountering an issue with your Retrospective Record script. It doesn't seem to work properly in my system. I have installed all required scripts including the 'Various_functions.lua' file and then:

1) I create a new track and call the action "Prepare selected track for Retrospective Record";
2) I create another track and add a VSTi;
3) I play some notes and trigger 'Dump Retrospective Record tracker log'.

However, nothing happens. Please see the link below:

https://imgur.com/a/3Ybj52X


I then try to add the 'Retrospective Record_tracker' JS plugin in the Input FX slot of that same track. Only a couple of random notes appear.

Here is the second link:

https://imgur.com/a/GfvbMtq#TCqP5pP


Could you please check if there is any bug there or is it me doing something wrong?

Thanks bro!

[mpl]

You have to enable correct MIDI input for prepared track and arm it once. It does designed to use only prepared track for storing data.

[Supremo]

Originally Posted by mpl

You have to enable correct MIDI input for prepared track and arm it once. It does designed to use only prepared track for storing data.

Sorry, Mikhail, but it doesn't seem to work properly. I tried so many various options: armed the prepared track, tried selecting different midi inputs, made the prepared track a parent to the instrument track, separated them, etc... All my attempts have failed.

When subordinating the instrument track to the prepared track only some random notes appear in the midi item with no sound.

UPD: The script happens to work correctly only with the virtual midi keyboard but not with the real keyboard.

[Stroudy]

I'm not getting any meaningful data when using 'Dump Retrospective Record tracker log'

Sometimes I don't get any data, other times an empty item, other times notes I didn't play. totally random.

I wish I could give more info, but it really is that random.

[ipopon]

Same here.

I've installed 3 scripts: Clear, Dump, Prepare retrospective record.

It worked about 2 times, but after that the script seemed to start working randomly and a bit later stopped working at all. Tried, restarting Reaper, clearing, repreparing tracks, but no results.

Also, when I run "Prepare retrospective" on a track, it unchecks Route/"Master Send" and sets "record: disable (input monitoring only)".

[paaltio]

Originally Posted by EvilDragon

Great work (as always) mpl, but man, wouldn't it be so much easier if this was just built into Reaper, no fussing with JSFX or anything?

Just chiming in here to point out Digital Performer v10.1 just added Retrospective Record and Cubase 10.5 recently had a big upgrade to its feature... *winks heavily in Justin and schwa’s direction*

Thanks for the great work mpl in the meantime!

[Yorkz]

Originally Posted by Supremo

UPD: The script happens to work correctly only with the virtual midi keyboard but not with the real keyboard.

Same. Works fine with virtual keyboard. Entering notes via midi keyboard returns a glitchy midi clip.

[mpl]

Not sure if it helps, but I removed offset decrement from JSFX code. So hopefully it will work better (but less precise). I guess I have to convert MIDI receive offset from PPQ to seconds, then decrement it from time precise, but I have bad JSFX skills to do it so I leave it as is.

[Yorkz]

Originally Posted by mpl

Not sure if it helps, but I removed offset decrement from JSFX code. So hopefully it will work better (but less precise).

Tried the new version and it works like a charm. Thanks!

[moonuel]

Hi MPL! Thanks for the great script.

I just wanted to report a minor bug that I noticed: when dumping the log to the arrange view under certain conditions, REAPER doesn't seem to be aware of the new MIDI item generated to contain the log until the arrange view is updated. I noticed this when trying to dump the log to the paused play cursor far enough into the project because the expected behaviour of playing the MIDI log doesn't occur - instead the play cursor jumps back to the beginning of the project.

Steps to reproduce:
Open a blank project file, create two new tracks, and prime one for retrospective recording using the appropriate action. Move the play cursor to the 1 minute marker and make sure the play cursor is stopped. Play some MIDI notes from an external controller (doesn't seem to matter how long the input string is). Dump the log to the second, unprimed track, creating a new MIDI item at the 1 minute mark. Press play. The play cursor should loop back to the beginning and not play the new MIDI item. This issue persists even if the play cursor is moved, but is resolved if the arrange view is updated by making a time selection, by making a new item, or by moving the MIDI log item.

Thanks again for the great work, hope this bug report is useful.

[Hellfog]

MPL:

Can you make a script, so when using "Script: mpl_Prepare selected track for RetrospectiveRecord tracker"


It is placed in "IN FX" window instead of Main FX window?

[tusitala]

is there a tutorial in order to use this?
Even if I'm a Reaper user for 10 years, I'm not very good with scripts and I've never done anything that looks this "complicated"

Apologise for my ignorance.

Regardless.

Thanks a lot for your work!
Best

[mpl]

Here is new approach to test for new introduced API (thanks to Justin), no need for JSFX or other additional shit, still early test though:
https://forum.cockos.com/showpost.ph...8&postcount=23

[mpl]

https://forum.cockos.com/showpost.ph...1&postcount=31

Added to Reapack as mpl_Dump retrospective record log lua

[Catesby]

oh super cool thank you!!

[Breeder]

Originally Posted by mpl

https://forum.cockos.com/showpost.ph...1&postcount=31

Added to Reapack as mpl_Dump retrospective record log lua

Thank you man, this is pure gold!


One question, am I going crazy or does the script insert previously inserted events?

For example, if you use retrospective record to insert first 4 notes you played and then play another 4 notes later and if you try to use retrospective record again, it will insert all 8 notes instead of the last 4 and not ignore the previous 4 notes that have already been inserted with first call to retrospective record?

[mpl]

Originally Posted by Breeder

Thank you man, this is pure gold!


One question, am I going crazy or does the script insert previously inserted events?

For example, if you use retrospective record to insert first 4 notes you played and then play another 4 notes later and if you try to use retrospective record again, it will insert all 8 notes instead of the last 4 and not ignore the previous 4 notes that have already been inserted with first call to retrospective record?

Yeah I probably rewrite it according to rc2 addition + filter older buffered stuff (using ext state somehow I guess).

[Breeder]

Originally Posted by mpl

Yeah I probably rewrite it according to rc2 addition + filter older buffered stuff (using ext state somehow I guess).

Thank you for the kind response!

[lb0]

Originally Posted by mpl

Yeah I probably rewrite it according to rc2 addition + filter older buffered stuff (using ext state somehow I guess).

Filtering old events is pretty straightforward. When reading the first event (ie. last played) - store the first return variable (retval) in ExtState (non-persistent). Then on subsequent captures - read this stored value and loop through all events while the retval is > stored return value.

I know you'll work out how to do this - just thought it may save you a few minutes...

[paaltio]

Also to raise one note from the Reaticulate thread: currently the script creates a take for each device index, which becomes a bit of a mess when you have like I do events coming in from the hardware MIDI input (keyboard), virtual inputs (TouchOSC) and VKB (Reaticulate)! So I would suggest combining the MIDI from all the device indexes to a single take, so the end result is the same as normal recording.

[ArtemiHo]

Hello!

So I've downloaded the latest MPL-Justin Retrospective script
but it seem to do nothing, when I click on the track nothing happens

The old way of inserting the jsfx script seem to work but the main script seem to remove the master send of the track for some reason..


So how exactly one should use it? are there any instructions?

[mpl]

I removed everything related to old version which used JSFX tracker.

Also, mpl_Dump Retrospective Record log.lua was updated and converted into metapackage contain following versions:

• mpl_Dump Retrospective Record log.lua - kinda general script, uses optimized settings, including breaks (don`t get previous added data) and obey looped recordings as multiple takes
• mpl_Dump Retrospective Record log (notes only).lua
• mpl_Dump Retrospective Record log (only data at playing).lua - event put into their played positions
• mpl_Dump Retrospective Record log (only data at stop).lua - start from edit cursor to a event list length
• mpl_Dump Retrospective Record log (everything recorded from last REAPER start).lua - collect everything without any filters
• mpl_Dump Retrospective Record log (everything from last 5 minutes).lua - ignore break, dump everything based on time limit
• mpl_Dump Retrospective Record log (everything from last 10 minutes).lua
• mpl_Dump Retrospective Record log (everything from last 30 minutes).lua
• mpl_Dump Retrospective Record log (everything from last hour, obey stored data break).lua

mpl_Dump Retrospective Record log v1 works for free (no dependency from VariousFunctions package) and mostly made around Justin`s code.
Version 2 have slightly different concept and different way of sorting/handling data. It uses VariousFunctions package 2.62+ (which is paid).
Also, the second version is easier to extend so any suggestions welcome.

However if anyone have serious bug report about mpl_Dump Retrospective Record log v1, I can make a free version of this script contains basic features.

Originally Posted by paaltio

Also to raise one note from the Reaticulate thread: currently the script creates a take for each device index, which becomes a bit of a mess when you have like I do events coming in from the hardware MIDI input (keyboard), virtual inputs (TouchOSC) and VKB (Reaticulate)! So I would suggest combining the MIDI from all the device indexes to a single take, so the end result is the same as normal recording.

This is done for v2. If someone needs a split by devices, I can add it.

Originally Posted by lb0

Filtering old events is pretty straightforward. When reading the first event (ie. last played) - store the first return variable (retval) in ExtState (non-persistent). Then on subsequent captures - read this stored value and loop through all events while the retval is > stored return value.
I know you'll work out how to do this - just thought it may save you a few minutes...

Yeah thats pretty how I did it. Version 2 handle both old events AND time limit. Like: get events from last break BUT which was stored in last 5-10-20-30 minutes.

[Arthur McArthur]

Great stuff, mpl. Just sent a donation. Thanks for this

[C-H]

I must be an idiot.
Trying to use the free version and get the window saying how to go 'back to free version'.
I follow the instructions but it says I need to upgrade to the (paid) 2.67 version.

I am obviously missing something here but what?

Thank you in advance.

[mpl]

Originally Posted by C-H

I must be an idiot.
Trying to use the free version and get the window saying how to go 'back to free version'.
I follow the instructions but it says I need to upgrade to the (paid) 2.67 version.

I am obviously missing something here but what?

Thank you in advance.

Ok it is not something obvious so I added
mpl_Dump Retrospective Record log (old version)
so it doesn`t collide with new version, it is another package. This script doesn`t require VariousFunctions package at all (as I mentioned it is rough code, the core stuff built by Justin, second version uses more advanced ways to handle data).

[mpl]

Originally Posted by ArtemiHo

Hello!

So I've downloaded the latest MPL-Justin Retrospective script
but it seem to do nothing, when I click on the track nothing happens

The old way of inserting the jsfx script seem to work but the main script seem to remove the master send of the track for some reason..


So how exactly one should use it? are there any instructions?

The script creates MIDI take on selected track (add new track if no selected track), put all incoming MIDI events into this take, sorted by device.
Sorry for inconvenience, I edited head post: https://forum.cockos.com/showpost.ph...41&postcount=1

[C-H]

Originally Posted by mpl

Ok it is not something obvious so I added
mpl_Dump Retrospective Record log (old version)
so it doesn`t collide with new version, it is another package. This script doesn`t require VariousFunctions package at all (as I mentioned it is rough code, the core stuff built by Justin, second version uses more advanced ways to handle data).

Thank you!
I'll check it out.

[C-H]

Originally Posted by mpl

Ok it is not something obvious so I added
mpl_Dump Retrospective Record log (old version)
so it doesn`t collide with new version, it is another package. This script doesn`t require VariousFunctions package at all (as I mentioned it is rough code, the core stuff built by Justin, second version uses more advanced ways to handle data).

Works great!
Thank you.