Better handling of high track counts and VEP

[EvilDragon]

OP is doing that already. Everybody working with such templates is already doing that.


Really the main problem here is the processing overhead that Reaper has on empty tracks, which gets very high when on some other DAWs there's no such processing overhead.

[Lokasenna]

Forgive the idiocy, but how are they using any notable CPU? I'm assuming Do not process muted tracks is checked.

[EvilDragon]

Try it yourself. Create a new project and just keep on adding empty tracks, get to a couple thousand, then see how much CPU is taken.

[Lokasenna]

Yeah, I see the issue - the idle CPU usage is all GUI overhead.

Consequently, hiding tracks in the TCP and/or MCP from the Track Manager makes a huge difference - you can also Solo and Mute from the Track Manager, so for the auditioning process that might be a useable workaround. i.e.:

- All extra tracks are completely hidden.
- Browse through them with the track manager, muting and unmuting as needed.
- When you decide you want one of them, unhide it.

With 1024 tracks, hiding them from just the MCP cut Reaper's reported CPU usage by about two thirds on my end.

[EricTbone]

Originally Posted by Lokasenna

the idle CPU usage is all GUI overhead

Click "Play".

[srdmusic]

Originally Posted by EvilDragon

It's not about 20 spot mics, it's about having a large number of sounds/instruments/articulations available at the drop of a hat...

Thanks for elaboratating ED. It's not that I even need 4000 track. Just 1000 would hold a very large template for me but even that is idleing at 45% CPU and the GUI/GPU is slow with all instruments offline and all tracks muted. That's a rediculious amount of CPU for reaper to use without actually processing any samples. I have a 48core with 128gb ram so if my system is struggling, most users will see a problem with track counts above 500 tracks.

[James HE]

Core 2 Quad with 4 GB ram. Window7 64 bit. 2200 tracks. very minimal cpu on playback





now, something strange happened before I took this screenshot. I was digging around preferences and saw an option:

Preferences > Audio > Reduce CPU use of silent tracks during playback (experimental)

(is this option new btw?)

So, remembering this thread, I checked it, and added ~2000 tracks and was getting 25% CPU usage at idle. GUI locked up for a while. hit play and cpu went up a little, REAPER stopped responding, i closed the window, twice, to see if it would start responding, then it crashed to desktop.

I then got a message from windows saying that Windows had detected a problem with REAPER and that some compatibility settings have been changed...

um what?

so not sure what this is about. Can't even figure out what if anything was actually changed.


the "silent tracks" option makes no difference here. I do get a little more CPU usage with ASIO than with WASAPI.

Did I break something? Did I fix something?

[karbomusic]

Originally Posted by James HE

I then got a message from windows saying that Windows had detected a problem with REAPER and that some compatibility settings have been changed...

um what?

Right-click reaper.exe > properties > compatibility and see what the settings are. It may be as simple as its reverting to run with an older OS compat setting or similar.

[James HE]

Originally Posted by karbomusic

Right-click reaper.exe > properties > compatibility and see what the settings are. It may be as simple as its reverting to run with an older OS compat setting or similar.

not running in compatibility mode.


when I added tracks, I held down CTRL+T till it added ~200 tracks or so, then hit CTRL+A, CTRL+C, then CTRL+V a bunch of times.

I think maybe this triggered the "sticky keys" accessibility notification - but since I basically hit the CPU with a pretty heavy task of adding infinite tracks, maybe this made every thing crash and I didn't even see that dialog???

I am getting really strange keyboard and mouse behaviors atm. i think I did turn on some acessability feature or something.


will shut down - will test on a portable REAPER install and see what's going on. Will have to do that later on.

[EvilDragon]

Sticky keys is Shift held down 5 seconds, not sure what you turned on there...

[James HE]

Originally Posted by EvilDragon

Sticky keys is Shift held down 5 seconds, not sure what you turned on there...

$%^& I dunno either. scroll wheel in my browser now zooms instead of scrolls, clicking on a link opens a new tab - what on earth, never had an app crash and cause such weirdness...

I'm pretty sure that is all unrelated to my performance here. In my first attempt at adding tons of tracks I simply broke something. If I take my time adding tracks, I can get up around 3200 empty tracks, and hit play and see about 7% CPU usage and ~25% RT CPU usage at 1024 buffer size. cpu usage doubles at 512. GUI is mostly unusable at 512. but going down to 2000 tracks and things are pretty good. (will test with some MIDI items and instruments later).

Some of you have 10x the CPU power that I have, so something isn't scaling like you'd think it should. Perhaps some of this may be related to hyperthreading. I have 4 non-hyperthreaded cores. Just a stab in the dark.

I will test more later.

To anyone who has trouble playing a project with about 2000 empty tracks, feel free to post or send me your REAPER config so i can compare - maybe there is some setting that helps in running high track counts. ??

[EricTbone]

Originally Posted by James HE

Core 2 Quad with 4 GB ram. Window7 64 bit. 2200 tracks. very minimal cpu on playback [..] Did I break something? Did I fix something?

CPU usage during playback on my machine:

2000 tracks = < 1%
4000 tracks = 40% and 50%

So perhaps there's some degenerative, exponential thing that happens somewhere in there, and perhaps the threshold differs depending on CPU, drivers, etc.

[James HE]

what CPU @EricTbone?

[EricTbone]

Originally Posted by James HE

what CPU @EricTbone?

i7-6820HQ (laptop)

I think the takeaway is that your success with 2K tracks doesn't necessarily mean anything people who are having this issue. Could be you're just under the threshold where the problem occurs for your machine.

[FnA]

Originally Posted by James HE

$%^& I dunno either. scroll wheel in my browser now zooms instead of scrolls, clicking on a link opens a new tab - what on earth, never had an app crash and cause such weirdness...

I was testing a zoom script used with mousewheel on a large number of audio items, because I noticed sluggishness in that scenario before, with regular mousewheel zoom. Hanged Reaper and lost use of my mouse altogether. Managed to shut down via task manager with keyboard. The mouse still wasn't working after Reaper was killed. Seems ok now, I think. This was a couple months ago. Windows 7x64.

[Eliseat]

Originally Posted by EricTbone

CPU usage during playback on my machine:

2000 tracks = < 1%
4000 tracks = 40% and 50%

So perhaps there's some degenerative, exponential thing that happens somewhere in there, and perhaps the threshold differs depending on CPU, drivers, etc.

That's exactly what I posted on the first page. In between 3000 - 4000 tracks Reaper raises the cpu usage exponentially. I guess there is something odd going on. But beside that its still strange that some guys here with power machines have weirder results then me with my old laptop.

[srdmusic]

I've found the following settings help very large sessions playback more stable:

Audio Preference: Reduce CPU use of silent tracks during playback (experimental) checked
Buffer Preference: Audio processing threads set to number of cores in my machine including virtual (48)
Buffer Preference: Thread Priority Highest, Behavior 0 - Relaxed
Track Preference: Allow media buffering and Anticipative FX on selected track

The only thing that's really slow in my 1000 track template is track selection when all tracks are visible.

What have you guys found to be the best setting for very high track counts?

[Travesty]

Buffer Preference: Thread Priority Highest, Behavior 0 - Relaxed

This seems to be the one for me. I've been setting up a large template, and before I set this, I was running at 30% cpu, with no fx enabled and around 60 empty tracks! There are around 60 sends in the project, though 36 of them are muted by default.

I previously had the behaviour set to Agressive.

I have yet to test the performance of the template with these settings.

This is on an 8700k OC to 4.4GHz with 32Gb of ram, using the intel on chip gpu.

[_Stevie_]

Awesome guys, thanks for the info. Keep it coming! I will also do some tests.

[Swi]

For anyone who was following this and also use large templates.
I have found using track templates is the answer for us. I started with my big template and saved every instrument and its accosiated midi tracks as track templates so the bussing and sends are all correct and now I start with a lean template and everything I need I import from the resource window that I keep docked on the left.
It works beautifully and I dont have thousands of unused tracks bogging me down.

I do wish that the track templates that I import appear only in the current track view screenset but I have asked for this feature now.

If anyone knows how to import a track template and have it only visible in the current track view screenset please let me know.

Thanks everyone.

[srdmusic]

[QUOTE=Swi;2041207
If anyone knows how to import a track template and have it only visible in the current track view screenset please let me know.

Thanks everyone.[/QUOTE]

I believe there is a way to hide all tracks with specific names like STR or BRS for stings and brass. If not there might be an easy script that can do it.

You might want to hit up Leon on the forum. I'll send you his info.

[storyteller]

Originally Posted by srdmusic

I believe there is a way to hide all tracks with specific names like STR or BRS for stings and brass. If not there might be an easy script that can do it.

You might want to hit up Leon on the forum. I'll send you his info.

OTR (Orchestral Template for Reaper) has this concept prebuilt into it. I also am currently experimenting with a donation model rather than the $129 regular price. You can check it out at http://otr.storyteller.im.

[Travesty]

Originally Posted by Swi

For anyone who was following this and also use large templates.
I have found using track templates is the answer for us. I started with my big template and saved every instrument and its accosiated midi tracks as track templates so the bussing and sends are all correct and now I start with a lean template and everything I need I import from the resource window that I keep docked on the left.
It works beautifully and I dont have thousands of unused tracks bogging me down.

Haha, yes I have ended up using the exact same approach.

[blumpy]

The reapack action:

Toggle visibility of Empty non-folder tracks matching '•'

is very handy for this purpose.

I put a '•' at the front of all midi tracks. If there's a midi item on a track with the bullet point in the name it will not hide. All midi tracks with bullet points in the name that is not record enabled and without a midi item will toggle show hide instantly.

This allows me to show all the tracks, arrow to the instrument I'd like to record, I hit the action via a key command, it hides everything else, and I can record and edit that track without the burden of a thousand blank tracks.

More simply put:
My template has a lot of midi tracks. I put the '•' at the front of all midi tracks. If a track does not have midi information on it, it will show/hide with using this action/script. If I've recorded midi on that track, it will remain visible when toggling the action/script. This makes track management for recorded midi and hiding unused midi tracks from a template very simple.

[Lokasenna]

Originally Posted by srdmusic

I believe there is a way to hide all tracks with specific names like STR or BRS for stings and brass. If not there might be an easy script that can do it.

You might want to hit up Leon on the forum. I'll send you his info.

I have "Show only specified tracks" on ReaPack, which lets you grab them by a search string as well as "children of ___", etc, apply it to the TCP or MCP or both, and save presets as separate actions so you can throw them on a toolbar or shortcut keys.

[blumpy]

Originally Posted by Lokasenna

I have "Show only specified tracks" on ReaPack, which lets you grab them by a search string as well as "children of ___", etc, apply it to the TCP or MCP or both, and save presets as separate actions so you can throw them on a toolbar or shortcut keys.

That looks great! A way to toggle just certain groups, (toggle the show hide state of groups, STR, BRS, etc) without hiding tracks containing midi items and without changing the show/hide state of other tracks would be cool, at least for me and my workflow. I like to keep busses and returns visible as well as video, DIA and SFX tracks.

[Travesty]

Originally Posted by Lokasenna

I have "Show only specified tracks" on ReaPack, which lets you grab them by a search string as well as "children of ___", etc, apply it to the TCP or MCP or both, and save presets as separate actions so you can throw them on a toolbar or shortcut keys.

This is excellent

[_Stevie_]

Originally Posted by blumpy

That looks great! A way to toggle just certain groups, (toggle the show hide state of groups, STR, BRS, etc) without hiding tracks containing midi items and without changing the show/hide state of other tracks would be cool, at least for me and my workflow. I like to keep busses and returns visible as well as video, DIA and SFX tracks.

@Loka this is a great script! I was about to code something similar, but no need anymore

However, I concur what blumpy says. It would be great if we could additionally *toggle* visibility of tracks.
So, we could have the option to show tracks exclussively and toggle tracks.

[blumpy]

Once I get to a couple hundred tracks its still sluggish. Showing only the tracks I'm working on makes it usable but the interface could improve significantly considering a large template is going to result in most tracks being empty and unused. I'm not sure how to resolve this.

[_Stevie_]

It's actually a Reaper issue. Reaper doesn't handle a huge amount of tracks very well, even if they are deactivated. I hope it can be improved in future versions.

[mschnell]

I suppose by "handle" you mean the GUI, not the audio engine.
What do you mean by "deactivated" ?

-Michael

[blumpy]

Yes the GUI. The only way to make it manageable is to hide most of the tracks but even then it’s slow and clumsy.

I love reaper but currently large templates are impossible to work with. Even importing midi from a template from another daw will bring reaper to its knees. It took me 2 hours just to delete unused/empty midi tracks from a DP session someone else sent me.

[storyteller]

Originally Posted by blumpy

Yes the GUI. The only way to make it manageable is to hide most of the tracks but even then it’s slow and clumsy.

I love reaper but currently large templates are impossible to work with. Even importing midi from a template from another daw will bring reaper to its knees. It took me 2 hours just to delete unused/empty midi tracks from a DP session someone else sent me.

I optimized the heck out of it from a settings standpoint in OTR and almost completely eliminated the GUI lag. For excessively large templates, there is a menu button in OTR that toggles auto-scroll for both playback and recording. I found these were the biggest culprits. So if you can optimize everything, then when you get really encumbered with high track counts all you have to do is toggle the auto-scroll feature.

Check out OTR (otr.storyteller.im) and see how it performs compared to your setup. The highly optimized reaper performance is one of the biggest and most immediate compliments I receive on it before users dive into workflow.

[mschnell]

Originally Posted by blumpy

Yes the GUI.

Is this on any OS and Hardware (PC/Mac/Linux) ?
I just bought (not for me) a "Gaming Laqptop" that (other than most Laptops) has an Nvidia GPU with it's own memory - and a 6/12 core i7 processor) that might make a difference regarding GUI performance ...

-Michael

[blumpy]

Originally Posted by mschnell

Is this on any OS and Hardware (PC/Mac/Linux) ?
I just bought (not for me) a "Gaming Laqptop" that (other than most Laptops) has an Nvidia GPU with it's own memory - and a 6/12 core i7 processor) that might make a difference regarding GUI performance ...

-Michael

Yes that is the case on any platform but remember it's only for large templates.

The GPU will make no difference.

[mschnell]

Originally Posted by blumpy

The GPU will make no difference.

Supposedly not. But the separate video memory might.

-Michael

[EvilDragon]

Probably not.

[blumpy]

Originally Posted by mschnell

Supposedly not. But the separate video memory might.

-Michael

I have several workstations (32/64 core-thread thru 6 core) all with discrete GPUs; Vega 64, 1080ti, 1070, 550.

If there is an improvement from dedicated graphic memory it's not enough to make a difference.

[TonE]

Why using so few tracks like 4000 and not 1000000, ah, 1 million?

Five ideas:
(a) Reaper could use some compressed internal format, like zip, and not its normal text representation which make those files huge.

(b) Reaper could use a database, something like sqlite, for storing those texts, in the hope it will speed up things. Any expert opinions?

(c) Reaper could save not a single .rpp, but instead multiple splitted components/parts, well, bestly would be just organizing in the mentioned sqlite db, then pulling from there whatever it needs.

(d) Reaper could do same as c but without a db, then storing again into separate text files or their zipped forms, inside a directory. So a single .rpp would be converted to a single directory with multiple components inside, as text or their zipped form. When loading a project Reaper would scan this directory content and build up its data. And auto-save would be automatically splitted into smaller components, so this should speed up things as well, let us hope.

(e) Reaper could use an sqlite db per project, so the directory approach as in c but per project. I guess this would be my favourite solution as projects are still indepently organized and sqlite is only there speeding up certain things, hopefully. Separation of components, zipped form rather than only text, sqlite supported, organized in a subdirectory, all this together could be the magic solution?

Testing could be done with 1 million tracks, so in future not another user coming with another monster scenario, like mixing 10 films together.

[tack]

Originally Posted by TonE

(a) Reaper could use some compressed internal format, like zip, and not its normal text representation which make those files huge.

I've often wished for this too. But I don't think the numbers bear it out.

Saving my ~150 track template to SSD takes about 2 seconds and the project size is 220MB on disk. (I suspect that's about 1s of I/O and 1s of serializing objects in memory.) Running that .rpp file through gzip --fast takes 4.8 seconds with a resulting file size of 136MB. (Default gzip compression takes 7.6s and results in a 128MB file so the time/space tradeoff is obvious.)

So that might reduce I/O time to 0.5s, does nothing for the serialization time, but then adds 4.8s to the mix, resulting in a ~3x increase in save time.

Looking instead at lz4, notorious for fast performance, compression takes 0.5s but results in a file size of 182MB. Coming from 220MB, that's scarcely worth the effort.

Originally Posted by TonE

(b) Reaper could use a database, something like sqlite, for storing those texts, in the hope it will speed up things. Any expert opinions?

This is where things are likely to help, IMO. Per-project sqlite files make sense because you can quite easily update only what's actually changed, rather than rewriting the entire file each time. Also you can write binary blobs directly and avoid the need to base64-encode all that data.

But now you're left with a binary file (a database even) which is harder to recover in case of corruption or bug. That said, sqlite3 has decent recovery tools.

Originally Posted by TonE

(c) Reaper could save not a single .rpp, but instead multiple splitted components/parts, well, bestly would be just organizing in the mentioned sqlite db, then pulling from there whatever it needs.

Then you're left with multiple files for your project. I admit I kinda like the single file artifact of reaper projects as I deal mostly with MIDI. If I dealt with a lot of audio, I suppose it'd not matter much as you're already used to managing multiple files in your project.

If it were me, I'd use an sqlite3 db per project. So we agree on the preferred option here.

But I don't think the developers have any interest in optimizing this. When I brought up the idea of compression, I seem to recall the text-based format being cited as the reason against. So the recovery/readability angle of the rpp format seems to be strongly favored.