Wishing for text-based audio & video editing feature

[cmacsound]

First feature request:

I work often with transcript-based material (for documentaries) and hope that one day REAPER could be configured to edit an interview via a text window, so that the corresponding audio items would follow the changes made in text.

I suppose the first step would be to have a phonetic text-align feature so that a transcript could be "imported" or overlaid on top of media items that contain its dialog. Then, one hopefully could edit some text to build a narrative, then perfect the timings and mix it!

This would save me months (maybe years) of my, and hopefully countless others' life.

Some big commercial, examples out there of something like this are ScriptSync, but something tells me that REAPER could do it way better, not to mention more stable...

Thanks for reading this.

[semiquaver]

this is a very interesting idea and could be implemented as a script I would imagine. Or a plugin.

[cmacsound]

Yes indeed! If only I knew how to code such a tool... !

Perhaps a JS plugin with a simple text interface, in conjunction using Gentle would be a step in the right direction:

https://github.com/lowerquality/gentle

[planetnine]

What editing do you want to do with this tool, specifically?

Can you give a mockup of an instruction set you might want to apply to an audio item? It might be possible in script.



>

[serr]

So... Take a recording of speech. Edit it as text. Push a button and presto-changeo you have a seamless edit of the speech audio with no artifacts?

There are some American TV "news" stations that would probably pay big money for that plugin!

[cmacsound]

Yes of course. So glad you think this may be possible to script.

Wishful workflow:

• Import multitrack or stereo mix interview into REAPER.
• Import transcribed interview dialog text into a REAPER JS script for automatic phonetic-syllable alignment, perhaps with the help of Gentle or something similar.

Here's where the magic would come in, after the phonetic alignment of the imported audio dialog and text:

• In a REAPER JS script text editor GUI, a user can edit, rearrange and discard entire sentences and words within the transcript just like one would edit text in a text editor AND have the media items automatically split and rearrange themselves in the TCP timeline, in a way that follows the newly edited structure of the text. (Essentially linking the media item audio to the transcript and vice versa).

• Close/hide the JS transcript editor window and make fine-tune media item edits manually in order to prep for a final mix.

• Reap huge success.

Ideally this script would be inserted an a parent folder track which contains the multitrack of the interview, i.e. host track, guest 1, guest 2, etc. embedded within the parent folder track. I think this way the user could have the ability to edit the script non-destructively, and make further changes in the storyline as the plot develops.

I see this as something that would be a huge game changer for all those producing stories and documentaries from scratch, both in audio and video.

Hope this helps. Please don't hesitate to ask if more clarification is needed!

[planetnine]

I'm following now.

X-Raym is your guy for subtitle actions and scripts, perhaps some of his stuff could be used for extracting edit-point data from the actual wording.

Personally I'd go for highlighting words in a script window and them being correspondingly highlighted in the items themselves for manual editing -you can tell that I don't trust automatic editing, can't you..?



>

[semiquaver]

you would certainly have to go back through all the edits but it would be great to have the script rough it out for you!

[X-Raym]

@cmacsound
That's an advanced request ^^

Meanwhile, I think you can consider just having your subtitle track with text item and make edit with Ripple Edit All Tracks ative,



Simple and easy solution. Not exactly what is asked, but it can help anyway I guess.
FOr reordering, set edit cursor at the begining of an item, select other items and make your copy paste with ripple active.

ALso, you can split item from first track according to items on track two and group items by columns (I have two scripts for that).

[cmacsound]

@X-Raym,

Yes, I realize this is an advanced request, and understand if I'm "shooting the moon" a little in asking for it...

I fully comprehend the text item based ripple edit idea, and thank you for the demonstration reminder. I suppose I'm looking for something that can handle a LOT more audio/text quickly to build a rough outline for script development.

And of course I'd be happy to donate to such a noble development!

@semiquaver,

Of course manual fine-tuning would always be necessary. But if up against countless hours of tape I can definitely see this option to be a HUGE advantage in building narratives and sketching out stories.

[planetnine]

I think using Raymonds subtitles to at least annotate your audio would be a start. Getting that workflow workable will be essential, whatever level you take your ideas to in the future.

Shooting for the moon is admirable, buts let's build the rocket first.

Can we get a speech recognition app to help with this..?



>

[cmacsound]

Yes, agreed.

I have my audio annotated (that's the "easy" part thanks to professional transcriptions. I also discovered that YouTube is a decent .srt aligner. For instance, If you upload a video and a transcript text, YouTube aligns all that up and lets you download a .srt file that we can import into REAPER which saves a bunch of time.

Here's what that can look like:

The next step is to move these text items' notes over to the media items' notes, but I can't seem to find a way to do that quickly and easily in a script just yet... ?

[X-Raym]

@cmacsound
PM sent :P

[heda]

Hi,
I've seen that Gentle is in Python. I don't know python. Everything could be possible. But sometimes it can be very difficult to do it. I'm not good enough to code something like this.
But it is nice that you can do at least the srt workflow with youtube.

[analogexplosions]

I'll be following this closely.

[cmacsound]

Thanks everyone for chiming in on this thread. Special thanks to X-Raym and HeDa for their very useful text note scripts!

Will continue to research this request in the coming months. For now, here's a site that is also trying to do this very thing, albeit with video on a web interface: http://hyperaud.io/

[cmacsound]

A recent REAPER update appears to already help moving things around in the project timelines a little.

Make a timeline selection, and right-click on the timeline. Some new options are available:

– Crop project to selection
– Insert empty space in selection
– Remove contents of selection (moving later items)

http://reaperblog.net/2016/07/quick-...removing-time/

[X-Raym]

@cmacsound


I found a text based editing software :



https://www.descript.com/


It has export to EDL, AAF, Premiere Pro etc... You could then do post prod in REAPER

[eviluess]

Originally Posted by X-Raym

@cmacsound
That's an advanced request ^^

Meanwhile, I think you can consider just having your subtitle track with text item and make edit with Ripple Edit All Tracks ative,



Simple and easy solution. Not exactly what is asked, but it can help anyway I guess.
FOr reordering, set edit cursor at the begining of an item, select other items and make your copy paste with ripple active.

ALso, you can split item from first track according to items on track two and group items by columns (I have two scripts for that).

hey, man, it's exiting to see this.
And how do we apply subtitle if we are editing a video all by reaper?
(the subtitle bruns into the video, not attached)

[X-Raym]

To burn subtitles in video in reaper,
you have to use several itemps with video processor FX and a text preset which would be able to hold our text and its display settings.


I have a script which can help pushing things as video processor text in bulk : https://www.extremraym.com/en/downlo...deo-processor/

[eviluess]

Originally Posted by X-Raym

To burn subtitles in video in reaper,
you have to use several itemps with video processor FX and a text preset which would be able to hold our text and its display settings.


I have a script which can help pushing things as video processor text in bulk : https://www.extremraym.com/en/downlo...deo-processor/

Good to see this in Reaper Channel, too.
I replied you in YouTube.

[hajomagre]

Hi,

I came into podcasting WAY after the start of this thread, but as a Reaper user I stumbled on it through a Google search.

I just watched a video from Riverside demonstrating their new text based editing functions. It would almost drive me to Riverside instead of Reaper for podcast editing.

https://www.youtube.com/watch?v=cEWJapLwqGI

But off course, Riverside is missing a lot that I use in Reaper to make audio sound great. Plugins to name one. And far better editing capabilities overall.

The question in this thread is still my question though. Can we have some sort of text based editing as demonstrated in that video?

Riverside says it does it by using AI for "unmatched accuracy". Could we make Reaper do the same? I am not a developer myself at all, but I can imagine that recent AI techniques would make a lot more possible than at the start of this thread in 2016.

To take it a step further: with more and more podcasting AND automatic transcriptions going on, would Reaper be missing the boat if it would not implement this in the near future?

Or did I just miss the right thread and is there some cool Reascript that does this for me? Would be more than happy to pay for it. Just point me in the redirection, please!

Note: I would be looking for something that supports Dutch, not English only.

Kind regards,
Hajo Magré

[Meo-Ada Mespotine]

Transcription isn't there yet and takes a lot of time when doing it with the most precise AI-training-model. Especially if it's not english or a popular language in particular.


With the Ultraschall-podcast extension for Reaper, we currently discuss how to add an "edit from text"-feature, but it's complicated to do.

I'm not aware of a solution already existing.

[hajomagre]

I was not aware of the existence of Ultraschall-podcast extension. Looking into it as we speak. So thanks for the reply anyways!

Will be monitoring this thread for future development on the topic of text based editing. Or text-aided, at least.

[Meo-Ada Mespotine]

Here's the official thread about Ultraschall:
https://forum.cockos.com/showthread.php?t=277354

[Ideosound]

Following this thread and hope there is a solution! 👀

[akademie]

I am also looking for Reaper's solution here 🤞 🙏 👂