Go Back   Cockos Incorporated Forums > REAPER Forums > REAPER Bug Reports

Reply
 
Thread Tools Display Modes
Old 06-25-2019, 02:57 PM   #1
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default Attention Developers : Reaper crashes when opening 2 project tabs

Hello Developers,

I am currently running a huge project with around 7000 files. I can't split this into multiple projects as this is one episode of a series. Off late I have been doing a lot of episodic work. I never get this problem when I am working on films as they are split into 20 minute reels. But episodes are big and the video is one single file. I am sound designing on this Netflix show currently which has 14 episodes in one season.

Now I want to have 2 or 3 episodes opened as tabs at the same time so that I can switch between projects and copy and paste between them (Especially things like backgrounds). These are some of the features why I switched to reaper in the first place from Protools.

I switched from protools world to reaper about 2 years ago...and I have to say its been an absolute blessing. Now I can't go back. I cannot even imagine what I would do without reaper.

I was speaking to Jon Tidey from reaper blog and he mentioned that there is some number of file limit that can be used by reaper. He did mention its somewhere around 8000 files on a mac...Now I understand why I am getting constant crashes and I even get error messages like "Too many files opened".



These days especially with netflix and amazon churning out series after series..episodic work has become more in number. So can something be done about this ?

I tried to set "Offline background project media" option in project tab when I switch between projects and it does work. But it takes a very long time to switch between projects as each project has nearly 7000+ files.


Here is a link to a screenshot from activity monitor. It shows the number of files used in a project. As you can see its using 6971 files. And believe me this is a light episode in terms of number of files...


https://drive.google.com/open?id=1gT...bQmY25hl3XjBOq


And here is a link to the project file...


https://drive.google.com/open?id=1D2...nhnEg6alP8W7Hn

I do have a powerful computer.

Here is a link to my computer spec


https://drive.google.com/open?id=1WI...MFhqHi9lx5R8aq

This requires some attention from developers pls...


Hope to hear from you soon
svijayrathinam is offline   Reply With Quote
Old 06-25-2019, 07:21 PM   #2
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

It does appear that the open file limit is kernel-imposed; to increase it you would have to (potentially) recompile or reconfigure the kernel.

Can you use subprojects to avoid having to keep so many files open? Or glue multiple files into one?
Justin is offline   Reply With Quote
Old 06-25-2019, 11:57 PM   #3
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Thank you for your response justin. I am already using Sub projects quite a lot. And I canít glue anything. I will loose handles and edits and access to the original media from the project. Most of the files are linked from different hard drives.

I wonder how protools is handling this. Because back in my protools days.. I used to do an entire season of a show in one session. I would keep the first episode at 1hr and 2nd ep at 2 hour and so on. Since protools canít open multiple projects like reaper.. I had to keep the entire season of the show in one episode and sure..it does have an advantage in working like that as you can copy paste from any episode., I am sure the number of files in that case will be very very huge.


Any ideas on how to recompile the kernel? Is it a simple process that I can try from my end? Or does it require scripting and coding knowledge?
svijayrathinam is offline   Reply With Quote
Old 06-26-2019, 02:13 PM   #4
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Hi justin , can you pls do something about this issue ? It’s killing me badly !! I wouldn’t be able to do this 14 episode season without this .. pls help
svijayrathinam is offline   Reply With Quote
Old 06-26-2019, 10:02 PM   #5
EvilDragon
Human being with feelings
 
EvilDragon's Avatar
 
Join Date: Jun 2009
Location: Croatia
Posts: 23,326
Default

https://superuser.com/questions/4337...on-os-x-10-7-1
EvilDragon is offline   Reply With Quote
Old 06-27-2019, 02:39 AM   #6
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Thank you so much evil dragon !!
svijayrathinam is offline   Reply With Quote
Old 06-27-2019, 02:42 AM   #7
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

While I was checking about how to solve this..I bumped into this feature request.

https://forum.cockos.com/showthread.php?t=203319

I think a feature like this would save so much of the computería resources plus it will prevent users from running into issues like this.

What do you think justin ?
svijayrathinam is offline   Reply With Quote
Old 06-27-2019, 06:10 AM   #8
Xenakios
Human being with feelings
 
Xenakios's Avatar
 
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 7,921
Default

Since all the thousands of files are likely not going to be accessed simultaneously, couldn't a mechanism be implemented to open and close the files dynamically as they are needed for playback or other purposes?
__________________
For info on SWS Reaper extension plugin (including Xenakios' previous extension/actions) :
http://www.sws-extension.org/
https://github.com/Jeff0S/sws
--
Xenakios blog (about HourGlass, Paul(X)Stretch and λ) :
http://xenakios.wordpress.com/
Xenakios is offline   Reply With Quote
Old 06-27-2019, 06:27 AM   #9
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

That would be brilliant. Not allocating Access for inactive takes would be a great start.
svijayrathinam is offline   Reply With Quote
Old 06-27-2019, 06:44 AM   #10
Xenakios
Human being with feelings
 
Xenakios's Avatar
 
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 7,921
Default

Another optimization for the number of open file handles could be to (optionally) keep the .reapeaks data entirely in memory. (I know about the existing memory mapped file option, but doesn't that keep the file handles open anyway?) Systems these days are going to have enough memory to handle that...
__________________
For info on SWS Reaper extension plugin (including Xenakios' previous extension/actions) :
http://www.sws-extension.org/
https://github.com/Jeff0S/sws
--
Xenakios blog (about HourGlass, Paul(X)Stretch and λ) :
http://xenakios.wordpress.com/
Xenakios is offline   Reply With Quote
Old 06-27-2019, 07:08 AM   #11
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Yes !! That would be brilliant!!
svijayrathinam is offline   Reply With Quote
Old 06-27-2019, 07:31 AM   #12
EvilDragon
Human being with feelings
 
EvilDragon's Avatar
 
Join Date: Jun 2009
Location: Croatia
Posts: 23,326
Default

Quote:
Originally Posted by Xenakios View Post
Since all the thousands of files are likely not going to be accessed simultaneously, couldn't a mechanism be implemented to open and close the files dynamically as they are needed for playback or other purposes?
Is that a real-time safe thing to do?
EvilDragon is offline   Reply With Quote
Old 06-27-2019, 07:39 AM   #13
Xenakios
Human being with feelings
 
Xenakios's Avatar
 
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 7,921
Default

Quote:
Originally Posted by EvilDragon View Post
Is that a real-time safe thing to do?
If the media buffering is enabled (as it should be), I would expect so. But even if once in a century some file doesn't manage to open and buffer up in time, it would be preferable to having crashes because files couldn't be opened at all...

edit : also, it could be provided as an option for "extreme" users, the default could be left to have the file handles open for all the files all the time.
__________________
For info on SWS Reaper extension plugin (including Xenakios' previous extension/actions) :
http://www.sws-extension.org/
https://github.com/Jeff0S/sws
--
Xenakios blog (about HourGlass, Paul(X)Stretch and λ) :
http://xenakios.wordpress.com/

Last edited by Xenakios; 06-27-2019 at 08:40 AM.
Xenakios is offline   Reply With Quote
Old 06-27-2019, 07:46 AM   #14
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Quote:
Originally Posted by Xenakios View Post
If the media buffering is enabled (as it should be), I would expect so. But even if once in a century some file doesn't manage to open and buffer up in time, it would be preferable to having crashes because files couldn't be opened at all...
Totally agree ��. I think dynamic file access allocation could be a huge performance improvement.
svijayrathinam is offline   Reply With Quote
Old 06-27-2019, 09:41 AM   #15
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Quick update : After evil dragon’s help I successfully managed to increase the number of file limit to 60,000. And now I am able to have 3 projects with close to 8000 files each open at the same time ! This is pretty good and it’s going to help me a lot..

So thank you very much evil dragon.

But I do notice reaper responding quite slowly on a powerful computer as mine. When I see performance meter.. I am only using 10GB out of 64 GB RAM and while playing total CPU usage is spiking up to 22% immediately as I hit play and then settles around 16%. But the video load spikes up to 400% on seek and play. It is during this time I am able to feel the slowness. And also while scrolling up and down during playback and also when idle. I have around 350 tracks.

So I am sure what xenokosis suggestions would help a great deal . Any more ideas on how to make this more efficient would be really appreciated. ��
svijayrathinam is offline   Reply With Quote
Old 07-05-2019, 08:08 AM   #16
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Hello Developers,

Inspite of changing the setting in kernel to access 65000 files, I am still getting "Too Many Files opened" issue especially when I open large projects or when opening multiple projects at the same time. Now I suspect its a reaper issue. Can you pls help me fix this ? How can this be solved. Pls help

Last edited by svijayrathinam; 07-05-2019 at 02:58 PM.
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 08:03 AM   #17
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default Files offline issue - Struggling to finish and delivery work...Developers Pls help

Hello Developers,


I have a huge project which has around 10,000 files. I can't glue or make subprojects here. As this is one episode of a 14 episode series. Each episode is 55 minutes long. when i open the reaper project..reaper does not say any files are missing. But when the project is opened..I see a lot of files say "Offline" even though the files are inside the project media directory.

Infact when I run the"SWS/S&M: Open selected item path in explorer/finder" action by selecting the file saying "offline"..it actually shows the file in finder. But for some reason reaper wouldn't play that file...And Justin said its a kernel issue on a mac. But after justin's suggestion..I set the kernel to allow 65000 files..But I am still getting this issue and when i hit play..it keeps saying...too many files opened errors..I am struggling very badly..I have a deadline and I would like to finish and deliver this project....Pls help me..

I have recorded a video and put it on youtube...


https://youtu.be/I6UahydTdZU


If you watch that video with audio turned up....you will hear me explain exactly what's going on....


I kindly request you guys to help me here...I am stranded....


Hoping to get a solution asap..pls


Here is a link to a project file


https://drive.google.com/file/d/16UP...ew?usp=sharing



And this is the error I am getting when I try to play the project or try to do anything...


https://drive.google.com/open?id=16_...m&usp=drive_fs

Last edited by svijayrathinam; 07-10-2019 at 08:11 AM.
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 09:10 AM   #18
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

Perhaps our detection from the OS of the hard file open count limit isn't working.

Making a test macOS build which will be on landoleet in about 20 minutes. The name will be 5.979rc1-lofc (large open file count). It hardcodes the internal limit to 40,000 open files. See if it helps!
Justin is offline   Reply With Quote
Old 07-10-2019, 09:17 AM   #19
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Thank you so much Justin !!
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 09:37 AM   #20
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

Posted! Let me know thanks!
Justin is offline   Reply With Quote
Old 07-10-2019, 09:55 AM   #21
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Yes.!! That solved it..All files are online...Thanks a Ton for such a quick fix


Meanwhile ..I just want to point out to other posts relating to this..and making reaper run large projects with ease...


https://forum.cockos.com/showthread.php?t=203319


https://forum.cockos.com/showthread.php?t=222807


X Ray M--helped me by making a script which offlines inactive takes..But the moment...I import a new file or render item as new take in existing project..it brings back all inactive offline takes back online...

Can this be solved ?


or like Xenakosis says.." Since all the thousands of files are likely not going to be accessed simultaneously, couldn't a mechanism be implemented to open and close the files dynamically as they are needed for playback or other purposes?"


I would think something like this will make reaper more powerful when working with large projects..wouldnt it ?
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 09:56 AM   #22
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Quote:
Originally Posted by Justin View Post
Perhaps our detection from the OS of the hard file open count limit isn't working.

Making a test macOS build which will be on landoleet in about 20 minutes. The name will be 5.979rc1-lofc (large open file count). It hardcodes the internal limit to 40,000 open files. See if it helps!
Also one more question..is it 40,000 files per project of reaper or 40,000 files across all opened projects ? (I would like to open at least 2 or 3 big projects in tabs at the same time so that i can switch between them and copy paste items between projects...
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 10:18 AM   #23
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

Thatís across all projects, however if you have the option set to offline background project media, that will help (offline media closes the file handles)
Justin is offline   Reply With Quote
Old 07-10-2019, 10:25 AM   #24
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

update: I do get some issues every now and then...Sometimes..when i zoom in a lot the waveform peaks disappear..and when i zoom back out..it becomes visible...Weird...Perhaps its not accessing the peak files properly ?
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 10:32 AM   #25
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

Also: for my reference, what macOS version is this? What process did you use to increase the max open file count?
Justin is offline   Reply With Quote
Old 07-10-2019, 10:33 AM   #26
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Quote:
Originally Posted by Justin View Post
Thatís across all projects, however if you have the option set to offline background project media, that will help (offline media closes the file handles)
this is exactly what I am doing currently...But with big projects such as these..it takes a very long time to get the project media back online..Its almost loser to project load times..which makes me feel like not having them open at all...And this defeats the purpose of very cool project tab feature... and I love having 2 or 3 projects opened at the same time...these are the features that made me switch from protools to reaper in the first place...


When I was working with Protools..I use to have all episodes in one big session..which slowed down my computer after a few episodes...(inspite of having a powerful computer)..But these project tabs in reaper is amazing..so we can only open 2 or 3 projects and just roll it with ease..If you can make the file offline and online process faster or perhaps some performance improvement that will go a very long way...
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 10:38 AM   #27
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Quote:
Originally Posted by Justin View Post
Also: for my reference, what macOS version is this? What process did you use to increase the max open file count?
i am on Mac os Mojave..here is a link to a screenshot of my system specs

https://drive.google.com/open?id=16a...m&usp=drive_fs

and the following is the command i used in terminal to increase the max file open count

"sudo launchctl limit maxfiles 65536 200000"


https://wilsonmar.github.io/maximum-limits/

the above link is what I followed to increase the max file open count
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 10:48 AM   #28
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Ok.. more update.. I am getting more and more too many files opened error.. especially when I am running render item as new take actions or some rendering scripts that X Ray m made or any scripts for that matter ..
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 11:27 AM   #29
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

Hmmm.

OK so I'm adding a new reaper.ini setting for 5.980rc2, which is (in the REAPER section):

Code:
osx_max_open_files=
You can specify a hardcoded value like:

Code:
osx_max_open_files=40000
or
Code:
osx_max_open_files=256000
And you can append a "v" to make it throw a messagebox up at startup to show you status, e.g.

Code:
osx_max_open_files=40000v
You can also use "a" as a value and it will try to autodetect (by trial and error) what the maximum value supported is.

Code:
osx_max_open_files=av
(remove the v if you decide you don't want to know).

So maybe try rc2 with osx_max_open_files=av, and try with something like:

Code:
sudo launchctl limit maxfiles 150000 250000
Also when you say "too many open files opened error", can you show me what that looks like?
Justin is offline   Reply With Quote
Old 07-10-2019, 12:26 PM   #30
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Quote:
Originally Posted by Justin View Post
Hmmm.

OK so I'm adding a new reaper.ini setting for 5.980rc2, which is (in the REAPER section):

Code:
osx_max_open_files=
You can specify a hardcoded value like:

Code:
osx_max_open_files=40000
or
Code:
osx_max_open_files=256000
And you can append a "v" to make it throw a messagebox up at startup to show you status, e.g.

Code:
osx_max_open_files=40000v
You can also use "a" as a value and it will try to autodetect (by trial and error) what the maximum value supported is.

Code:
osx_max_open_files=av
(remove the v if you decide you don't want to know).

So maybe try rc2 with osx_max_open_files=av, and try with something like:

Code:
sudo launchctl limit maxfiles 150000 250000
Also when you say "too many open files opened error", can you show me what that looks like?

Thank you so much Justin


Here is a link to the screenshot of how the error looks like

https://drive.google.com/open?id=16_...m&usp=drive_fs
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 12:39 PM   #31
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Than you so much Justin for your quick support. With your fix.. I managed to finish the first episode and I am making deliveries as we speak.. fingers crossed for RC2 .. I wish it solves the issue permanently.. Still 13 episodes to go for me ����
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 02:28 PM   #32
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Quote:
Originally Posted by Justin View Post
Hmmm.


Code:
osx_max_open_files=av
where do i add this code in the reaper.ini file? In which line exactly ? pls help
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 02:58 PM   #33
EvilDragon
Human being with feelings
 
EvilDragon's Avatar
 
Join Date: Jun 2009
Location: Croatia
Posts: 23,326
Default

It was mentioned - in REAPER section, so look for [REAPER]
EvilDragon is offline   Reply With Quote
Old 07-10-2019, 02:59 PM   #34
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

Quote:
Originally Posted by svijayrathinam View Post
where do i add this code in the reaper.ini file? In which line exactly ? pls help
under [REAPER]
Justin is offline   Reply With Quote
Old 07-10-2019, 03:05 PM   #35
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Quote:
Originally Posted by Justin View Post
under [REAPER]
Yes.. I found that. But can I add that at any line in reaper section or does it have to be at a particular line ?
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 03:13 PM   #36
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

Quote:
Originally Posted by svijayrathinam View Post
Yes.. I found that. But can I add that at any line in reaper section or does it have to be at a particular line ?
any line in that section..
Justin is offline   Reply With Quote
Old 07-10-2019, 03:26 PM   #37
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

Ok..I did what you suggested....


Code:
osx_max_open_files=av

Code:
sudo launchctl limit maxfiles 150000 250000



But it has made reaper very very slow..It wasn't like this in the 5.979rc1-lofc version...On that version I managed to finish an episode...Now I am opening the same project..but its very very slow..takes for ever to even do a save operation after seeking to a position in video
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 03:35 PM   #38
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

Quote:
Originally Posted by svijayrathinam View Post
Ok..I did what you suggested....


Code:
osx_max_open_files=av

Code:
sudo launchctl limit maxfiles 150000 250000



But it has made reaper very very slow..It wasn't like this in the 5.979rc1-lofc version...On that version I managed to finish an episode...Now I am opening the same project..but its very very slow..takes for ever to even do a save operation after seeking to a position in video
OK so try the old way:

Code:
sudo launchctl limit maxfiles 65536 200000
and (modifying the existing line that you added):
Code:
osx_max_open_files=40000v
(that should behave exactly like the 5.979rc1-lofc)

or
Code:
osx_max_open_files=60000v
(should give you an extra 20,000 handles)

and see if that helps?

Might be worth testing to see if it is the higher "launchctl" limit or the "osx_max_open_files" limit that makes things slow?
Justin is offline   Reply With Quote
Old 07-10-2019, 04:25 PM   #39
svijayrathinam
Human being with feelings
 
Join Date: May 2017
Posts: 544
Default

ok..I tried all different settings...I am sure that "osx_max_open_files" is what slowing things down. Upto 40,000 is ok..If I go any higher..it is slowing reaper down..very much...I actually installed previous version of reaper and increased launchtl limit to 1 million...and everything ran fast but the computer crashed..when I am in the middle of playback..But everything was fast when it lasted..

does this information help ? Or do you want me to try something else ?
svijayrathinam is offline   Reply With Quote
Old 07-10-2019, 04:27 PM   #40
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 12,435
Default

Not sure what I can really do beyond this -- if you set REAPER and the system to 65536 files, does that get really slow?
Justin is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 10:11 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.