|
|
|
06-25-2019, 02:57 PM
|
#1
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Attention Developers : Reaper crashes when opening 2 project tabs
Hello Developers,
I am currently running a huge project with around 7000 files. I can't split this into multiple projects as this is one episode of a series. Off late I have been doing a lot of episodic work. I never get this problem when I am working on films as they are split into 20 minute reels. But episodes are big and the video is one single file. I am sound designing on this Netflix show currently which has 14 episodes in one season.
Now I want to have 2 or 3 episodes opened as tabs at the same time so that I can switch between projects and copy and paste between them (Especially things like backgrounds). These are some of the features why I switched to reaper in the first place from Protools.
I switched from protools world to reaper about 2 years ago...and I have to say its been an absolute blessing. Now I can't go back. I cannot even imagine what I would do without reaper.
I was speaking to Jon Tidey from reaper blog and he mentioned that there is some number of file limit that can be used by reaper. He did mention its somewhere around 8000 files on a mac...Now I understand why I am getting constant crashes and I even get error messages like "Too many files opened".
These days especially with netflix and amazon churning out series after series..episodic work has become more in number. So can something be done about this ?
I tried to set "Offline background project media" option in project tab when I switch between projects and it does work. But it takes a very long time to switch between projects as each project has nearly 7000+ files.
Here is a link to a screenshot from activity monitor. It shows the number of files used in a project. As you can see its using 6971 files. And believe me this is a light episode in terms of number of files...
https://drive.google.com/open?id=1gT...bQmY25hl3XjBOq
And here is a link to the project file...
https://drive.google.com/open?id=1D2...nhnEg6alP8W7Hn
I do have a powerful computer.
Here is a link to my computer spec
https://drive.google.com/open?id=1WI...MFhqHi9lx5R8aq
This requires some attention from developers pls...
Hope to hear from you soon
|
|
|
06-25-2019, 07:21 PM
|
#2
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
It does appear that the open file limit is kernel-imposed; to increase it you would have to (potentially) recompile or reconfigure the kernel.
Can you use subprojects to avoid having to keep so many files open? Or glue multiple files into one?
|
|
|
06-25-2019, 11:57 PM
|
#3
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Thank you for your response justin. I am already using Sub projects quite a lot. And I can’t glue anything. I will loose handles and edits and access to the original media from the project. Most of the files are linked from different hard drives.
I wonder how protools is handling this. Because back in my protools days.. I used to do an entire season of a show in one session. I would keep the first episode at 1hr and 2nd ep at 2 hour and so on. Since protools can’t open multiple projects like reaper.. I had to keep the entire season of the show in one episode and sure..it does have an advantage in working like that as you can copy paste from any episode., I am sure the number of files in that case will be very very huge.
Any ideas on how to recompile the kernel? Is it a simple process that I can try from my end? Or does it require scripting and coding knowledge?
|
|
|
06-26-2019, 02:13 PM
|
#4
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Hi justin , can you pls do something about this issue ? It’s killing me badly !! I wouldn’t be able to do this 14 episode season without this .. pls help
|
|
|
06-26-2019, 10:02 PM
|
#5
|
Human being with feelings
Join Date: Jun 2009
Location: Croatia
Posts: 24,790
|
|
|
|
06-27-2019, 02:39 AM
|
#6
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Thank you so much evil dragon !!
|
|
|
06-27-2019, 02:42 AM
|
#7
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
While I was checking about how to solve this..I bumped into this feature request.
https://forum.cockos.com/showthread.php?t=203319
I think a feature like this would save so much of the computer’a resources plus it will prevent users from running into issues like this.
What do you think justin ?
|
|
|
06-27-2019, 06:10 AM
|
#8
|
Human being with feelings
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 8,062
|
Since all the thousands of files are likely not going to be accessed simultaneously, couldn't a mechanism be implemented to open and close the files dynamically as they are needed for playback or other purposes?
__________________
I am no longer part of the REAPER community. Please don't contact me with any REAPER-related issues.
|
|
|
06-27-2019, 06:27 AM
|
#9
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
That would be brilliant. Not allocating Access for inactive takes would be a great start.
|
|
|
06-27-2019, 06:44 AM
|
#10
|
Human being with feelings
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 8,062
|
Another optimization for the number of open file handles could be to (optionally) keep the .reapeaks data entirely in memory. (I know about the existing memory mapped file option, but doesn't that keep the file handles open anyway?) Systems these days are going to have enough memory to handle that...
__________________
I am no longer part of the REAPER community. Please don't contact me with any REAPER-related issues.
|
|
|
06-27-2019, 07:08 AM
|
#11
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Yes !! That would be brilliant!!
|
|
|
06-27-2019, 07:31 AM
|
#12
|
Human being with feelings
Join Date: Jun 2009
Location: Croatia
Posts: 24,790
|
Quote:
Originally Posted by Xenakios
Since all the thousands of files are likely not going to be accessed simultaneously, couldn't a mechanism be implemented to open and close the files dynamically as they are needed for playback or other purposes?
|
Is that a real-time safe thing to do?
|
|
|
06-27-2019, 07:39 AM
|
#13
|
Human being with feelings
Join Date: Feb 2007
Location: Oulu, Finland
Posts: 8,062
|
Quote:
Originally Posted by EvilDragon
Is that a real-time safe thing to do?
|
If the media buffering is enabled (as it should be), I would expect so. But even if once in a century some file doesn't manage to open and buffer up in time, it would be preferable to having crashes because files couldn't be opened at all...
edit : also, it could be provided as an option for "extreme" users, the default could be left to have the file handles open for all the files all the time.
__________________
I am no longer part of the REAPER community. Please don't contact me with any REAPER-related issues.
Last edited by Xenakios; 06-27-2019 at 08:40 AM.
|
|
|
06-27-2019, 07:46 AM
|
#14
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Quote:
Originally Posted by Xenakios
If the media buffering is enabled (as it should be), I would expect so. But even if once in a century some file doesn't manage to open and buffer up in time, it would be preferable to having crashes because files couldn't be opened at all...
|
Totally agree . I think dynamic file access allocation could be a huge performance improvement.
|
|
|
06-27-2019, 09:41 AM
|
#15
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Quick update : After evil dragon’s help I successfully managed to increase the number of file limit to 60,000. And now I am able to have 3 projects with close to 8000 files each open at the same time ! This is pretty good and it’s going to help me a lot..
So thank you very much evil dragon.
But I do notice reaper responding quite slowly on a powerful computer as mine. When I see performance meter.. I am only using 10GB out of 64 GB RAM and while playing total CPU usage is spiking up to 22% immediately as I hit play and then settles around 16%. But the video load spikes up to 400% on seek and play. It is during this time I am able to feel the slowness. And also while scrolling up and down during playback and also when idle. I have around 350 tracks.
So I am sure what xenokosis suggestions would help a great deal . Any more ideas on how to make this more efficient would be really appreciated. ��
|
|
|
07-05-2019, 08:08 AM
|
#16
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Hello Developers,
Inspite of changing the setting in kernel to access 65000 files, I am still getting "Too Many Files opened" issue especially when I open large projects or when opening multiple projects at the same time. Now I suspect its a reaper issue. Can you pls help me fix this ? How can this be solved. Pls help
Last edited by svijayrathinam; 07-05-2019 at 02:58 PM.
|
|
|
07-10-2019, 08:03 AM
|
#17
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Files offline issue - Struggling to finish and delivery work...Developers Pls help
Hello Developers,
I have a huge project which has around 10,000 files. I can't glue or make subprojects here. As this is one episode of a 14 episode series. Each episode is 55 minutes long. when i open the reaper project..reaper does not say any files are missing. But when the project is opened..I see a lot of files say "Offline" even though the files are inside the project media directory.
Infact when I run the"SWS/S&M: Open selected item path in explorer/finder" action by selecting the file saying "offline"..it actually shows the file in finder. But for some reason reaper wouldn't play that file...And Justin said its a kernel issue on a mac. But after justin's suggestion..I set the kernel to allow 65000 files..But I am still getting this issue and when i hit play..it keeps saying...too many files opened errors..I am struggling very badly..I have a deadline and I would like to finish and deliver this project....Pls help me..
I have recorded a video and put it on youtube...
https://youtu.be/I6UahydTdZU
If you watch that video with audio turned up....you will hear me explain exactly what's going on....
I kindly request you guys to help me here...I am stranded....
Hoping to get a solution asap..pls
Here is a link to a project file
https://drive.google.com/file/d/16UP...ew?usp=sharing
And this is the error I am getting when I try to play the project or try to do anything...
https://drive.google.com/open?id=16_...m&usp=drive_fs
Last edited by svijayrathinam; 07-10-2019 at 08:11 AM.
|
|
|
07-10-2019, 09:10 AM
|
#18
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
Perhaps our detection from the OS of the hard file open count limit isn't working.
Making a test macOS build which will be on landoleet in about 20 minutes. The name will be 5.979rc1-lofc (large open file count). It hardcodes the internal limit to 40,000 open files. See if it helps!
|
|
|
07-10-2019, 09:17 AM
|
#19
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Thank you so much Justin !!
|
|
|
07-10-2019, 09:37 AM
|
#20
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
Posted! Let me know thanks!
|
|
|
07-10-2019, 09:55 AM
|
#21
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Yes.!! That solved it..All files are online...Thanks a Ton for such a quick fix
Meanwhile ..I just want to point out to other posts relating to this..and making reaper run large projects with ease...
https://forum.cockos.com/showthread.php?t=203319
https://forum.cockos.com/showthread.php?t=222807
X Ray M--helped me by making a script which offlines inactive takes..But the moment...I import a new file or render item as new take in existing project..it brings back all inactive offline takes back online...
Can this be solved ?
or like Xenakosis says.." Since all the thousands of files are likely not going to be accessed simultaneously, couldn't a mechanism be implemented to open and close the files dynamically as they are needed for playback or other purposes?"
I would think something like this will make reaper more powerful when working with large projects..wouldnt it ?
|
|
|
07-10-2019, 09:56 AM
|
#22
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Quote:
Originally Posted by Justin
Perhaps our detection from the OS of the hard file open count limit isn't working.
Making a test macOS build which will be on landoleet in about 20 minutes. The name will be 5.979rc1-lofc (large open file count). It hardcodes the internal limit to 40,000 open files. See if it helps!
|
Also one more question..is it 40,000 files per project of reaper or 40,000 files across all opened projects ? (I would like to open at least 2 or 3 big projects in tabs at the same time so that i can switch between them and copy paste items between projects...
|
|
|
07-10-2019, 10:18 AM
|
#23
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
That’s across all projects, however if you have the option set to offline background project media, that will help (offline media closes the file handles)
|
|
|
07-10-2019, 10:25 AM
|
#24
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
update: I do get some issues every now and then...Sometimes..when i zoom in a lot the waveform peaks disappear..and when i zoom back out..it becomes visible...Weird...Perhaps its not accessing the peak files properly ?
|
|
|
07-10-2019, 10:32 AM
|
#25
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
Also: for my reference, what macOS version is this? What process did you use to increase the max open file count?
|
|
|
07-10-2019, 10:33 AM
|
#26
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Quote:
Originally Posted by Justin
That’s across all projects, however if you have the option set to offline background project media, that will help (offline media closes the file handles)
|
this is exactly what I am doing currently...But with big projects such as these..it takes a very long time to get the project media back online..Its almost loser to project load times..which makes me feel like not having them open at all...And this defeats the purpose of very cool project tab feature... and I love having 2 or 3 projects opened at the same time...these are the features that made me switch from protools to reaper in the first place...
When I was working with Protools..I use to have all episodes in one big session..which slowed down my computer after a few episodes...(inspite of having a powerful computer)..But these project tabs in reaper is amazing..so we can only open 2 or 3 projects and just roll it with ease..If you can make the file offline and online process faster or perhaps some performance improvement that will go a very long way...
|
|
|
07-10-2019, 10:38 AM
|
#27
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Quote:
Originally Posted by Justin
Also: for my reference, what macOS version is this? What process did you use to increase the max open file count?
|
i am on Mac os Mojave..here is a link to a screenshot of my system specs
https://drive.google.com/open?id=16a...m&usp=drive_fs
and the following is the command i used in terminal to increase the max file open count
"sudo launchctl limit maxfiles 65536 200000"
https://wilsonmar.github.io/maximum-limits/
the above link is what I followed to increase the max file open count
|
|
|
07-10-2019, 10:48 AM
|
#28
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Ok.. more update.. I am getting more and more too many files opened error.. especially when I am running render item as new take actions or some rendering scripts that X Ray m made or any scripts for that matter ..
|
|
|
07-10-2019, 11:27 AM
|
#29
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
Hmmm.
OK so I'm adding a new reaper.ini setting for 5.980rc2, which is (in the REAPER section):
Code:
osx_max_open_files=
You can specify a hardcoded value like:
Code:
osx_max_open_files=40000
or
Code:
osx_max_open_files=256000
And you can append a "v" to make it throw a messagebox up at startup to show you status, e.g.
Code:
osx_max_open_files=40000v
You can also use "a" as a value and it will try to autodetect (by trial and error) what the maximum value supported is.
Code:
osx_max_open_files=av
(remove the v if you decide you don't want to know).
So maybe try rc2 with osx_max_open_files=av, and try with something like:
Code:
sudo launchctl limit maxfiles 150000 250000
Also when you say "too many open files opened error", can you show me what that looks like?
|
|
|
07-10-2019, 12:26 PM
|
#30
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Quote:
Originally Posted by Justin
Hmmm.
OK so I'm adding a new reaper.ini setting for 5.980rc2, which is (in the REAPER section):
Code:
osx_max_open_files=
You can specify a hardcoded value like:
Code:
osx_max_open_files=40000
or
Code:
osx_max_open_files=256000
And you can append a "v" to make it throw a messagebox up at startup to show you status, e.g.
Code:
osx_max_open_files=40000v
You can also use "a" as a value and it will try to autodetect (by trial and error) what the maximum value supported is.
Code:
osx_max_open_files=av
(remove the v if you decide you don't want to know).
So maybe try rc2 with osx_max_open_files=av, and try with something like:
Code:
sudo launchctl limit maxfiles 150000 250000
Also when you say "too many open files opened error", can you show me what that looks like?
|
Thank you so much Justin
Here is a link to the screenshot of how the error looks like
https://drive.google.com/open?id=16_...m&usp=drive_fs
|
|
|
07-10-2019, 12:39 PM
|
#31
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Than you so much Justin for your quick support. With your fix.. I managed to finish the first episode and I am making deliveries as we speak.. fingers crossed for RC2 .. I wish it solves the issue permanently.. Still 13 episodes to go for me
|
|
|
07-10-2019, 02:28 PM
|
#32
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Quote:
Originally Posted by Justin
Hmmm.
Code:
osx_max_open_files=av
|
where do i add this code in the reaper.ini file? In which line exactly ? pls help
|
|
|
07-10-2019, 02:58 PM
|
#33
|
Human being with feelings
Join Date: Jun 2009
Location: Croatia
Posts: 24,790
|
It was mentioned - in REAPER section, so look for [REAPER]
|
|
|
07-10-2019, 02:59 PM
|
#34
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
Quote:
Originally Posted by svijayrathinam
where do i add this code in the reaper.ini file? In which line exactly ? pls help
|
under [REAPER]
|
|
|
07-10-2019, 03:05 PM
|
#35
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Quote:
Originally Posted by Justin
under [REAPER]
|
Yes.. I found that. But can I add that at any line in reaper section or does it have to be at a particular line ?
|
|
|
07-10-2019, 03:13 PM
|
#36
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
Quote:
Originally Posted by svijayrathinam
Yes.. I found that. But can I add that at any line in reaper section or does it have to be at a particular line ?
|
any line in that section..
|
|
|
07-10-2019, 03:26 PM
|
#37
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
Ok..I did what you suggested....
Code:
osx_max_open_files=av
Code:
sudo launchctl limit maxfiles 150000 250000
But it has made reaper very very slow..It wasn't like this in the 5.979rc1-lofc version...On that version I managed to finish an episode...Now I am opening the same project..but its very very slow..takes for ever to even do a save operation after seeking to a position in video
|
|
|
07-10-2019, 03:35 PM
|
#38
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
Quote:
Originally Posted by svijayrathinam
Ok..I did what you suggested....
Code:
osx_max_open_files=av
Code:
sudo launchctl limit maxfiles 150000 250000
But it has made reaper very very slow..It wasn't like this in the 5.979rc1-lofc version...On that version I managed to finish an episode...Now I am opening the same project..but its very very slow..takes for ever to even do a save operation after seeking to a position in video
|
OK so try the old way:
Code:
sudo launchctl limit maxfiles 65536 200000
and (modifying the existing line that you added):
Code:
osx_max_open_files=40000v
(that should behave exactly like the 5.979rc1-lofc)
or
Code:
osx_max_open_files=60000v
(should give you an extra 20,000 handles)
and see if that helps?
Might be worth testing to see if it is the higher "launchctl" limit or the "osx_max_open_files" limit that makes things slow?
|
|
|
07-10-2019, 04:25 PM
|
#39
|
Human being with feelings
Join Date: May 2017
Posts: 981
|
ok..I tried all different settings...I am sure that "osx_max_open_files" is what slowing things down. Upto 40,000 is ok..If I go any higher..it is slowing reaper down..very much...I actually installed previous version of reaper and increased launchtl limit to 1 million...and everything ran fast but the computer crashed..when I am in the middle of playback..But everything was fast when it lasted..
does this information help ? Or do you want me to try something else ?
|
|
|
07-10-2019, 04:27 PM
|
#40
|
Administrator
Join Date: Jan 2005
Location: NYC
Posts: 15,721
|
Not sure what I can really do beyond this -- if you set REAPER and the system to 65536 files, does that get really slow?
|
|
|
Thread Tools |
|
Display Modes |
Linear Mode
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
All times are GMT -7. The time now is 05:37 PM.
|