Save as new project with media files bug

[svijayrathinam]

I am trying to do a save project as with media files of a large project with about 16GB of media files..but it takes for ever. It has taken so far 6 hours and it still hasn’t finished doing it. The project contains various types of media items such as video files, flac files, wav files subprojects etc. the project also contains media files with different sample rates such as 192khz, 96 kHz, 48, 44.1. The project also has files referenced from other hard drives.

I understand a16gb project can take time..but definitely not this long. I have tried to do it in 3 different computers and i have the same problem in all of them. All computers are macintosh. Two computers running Mojave and one running high Sierra.

The problem exists in 5.965 version and all the pre releases that's going on currently. I have also posted this in the pre release section in this forum.

https://forum.cockos.com/showthread.php?t=216371

[Meo-Ada Mespotine]

What is it doing, still loading or rather calculating peaks?

You could try the following test: open a new project and import all the media files you use in the problematic project.

See, how long this takes and if there are any problems. That way we could rule out fileaccess problems.

(You don't need this testproject afterwards anymore.)

Oh and do you have older versions of the project? Do they have the same problem or did it appear at some point?


And which effects/plugins do you use in the problematic project? And Mac, Linux or Windows?

[svijayrathinam]

Originally Posted by mespotine

What is it doing, still loading or rather calculating peaks?

It says copying and the files are getting copied for ever !!

Originally Posted by mespotine

You could try the following test: open a new project and import all the media files you use in the problematic project.

See, how long this takes and if there are any problems. That way we could rule out fileaccess problems.

I tried that already. It makes no difference. It takes for ever !!

Originally Posted by mespotine

Oh and do you have older versions of the project? Do they have the same problem or did it appear at some point?

I tried it with many projects..even projects which doesn't have any files linked. I have a foley recording project in which all the files are recorded on that. Its still very very slow to copy. Its actually really fast to render all files and crop to active take and deliver. But I don't want to do this. I want the person who is receiving to be able to see all the take fx and stuff that I am using.

Originally Posted by mespotine

And which effects/plugins do you use in the problematic project? And Mac, Linux or Windows?

I am certain this has nothing to do with the plugins as it happens even in a foley recording project which has no plugins. But the big project I was trying to copy has the following plugins.

1. Exponential Audio R2 surround reverb
2. Altiverb XL
3. Fab Filter Pro Q3
4. Fab Filter Pro Q2
5. Waves H Reverb
6. Valhalla Room Reverb
7. IOSONO Anymix Pro
8. McDSP Futzbox
9. McDSP Dialogue Processor
10. Fab Pro R
11. Waves LFE 360

I have done this type of copying many times before..It worked fine before in the same computers. It only started happening recently. I dont normally deliver reaper projects so I didn't have the need to copy reaper project with media. But now I am collaborating with another reaper user hence the need. I am not entirely sure in which version this bug came in.

[karbomusic]

How many files? IOW, 16GB of small files (thousands and thousands) will take forever to copy compared to a single 16GB file - because each file has meta data to deal with. What I'm explaining generally lies outside of Reaper and is just about the quantity of files.

If you try to copy the files as a test using just the OS (outside of reaper), does that go slow?

[svijayrathinam]

Originally Posted by karbomusic

How many files? IOW, 16GB of small files (thousands and thousands) will take forever to copy compared to a single 16GB file - because each file has meta data to deal with. What I'm explaining generally lies outside of Reaper and is just about the quantity of files.

If you try to copy the files as a test using just the OS (outside of reaper), does that go slow?

The Project that I am trying to copy now is 32 GB. Its one full 45 minute episode foley recording project. It contains 3650 files. If I try to copy the entire folder..It takes around 15 Minutes. If reaper takes even 25 minutes to 30 minutes..I can totally understand. But it takes more than 6 hours. Surely there is something wrong. Pls help

[Meo-Ada Mespotine]

Originally Posted by svijayrathinam

I have done this type of copying many times before..It worked fine before in the same computers. It only started happening recently.

...

I am not entirely sure in which version this bug came in.

Ok, that's a hint. The best thing would be now to install several older Reaper-versions as portable install.
There's an option for portable install in the installer, so check it and select a folder DIFFERENT from your current installation and install Reaper.
Try to load the project in each of them, until you've found a working version.
If you've found the last Reaper version that loaded the project as fast as you remember, give us the Reaper-version.

Maybe something changed in the way Reaper works and we can find a setting that helps you.
Or it's a bug that happened somehow inbetween these versions.



3600 files should take some time, but not forever to open/import according to my experience with Reaper.

[svijayrathinam]

Originally Posted by mespotine

Ok, that's a hint. The best thing would be now to install several older Reaper-versions as portable install.
There's an option for portable install in the installer, so check it and select a folder DIFFERENT from your current installation and install Reaper.
Try to load the project in each of them, until you've found a working version.
If you've found the last Reaper version that loaded the project as fast as you remember, give us the Reaper-version.

Maybe something changed in the way Reaper works and we can find a setting that helps you.
Or it's a bug that happened somehow inbetween these versions.



3600 files should take some time, but not forever to open/import according to my experience with Reaper.

EDIT !!!

OK..Just went as far back as Ver 5.71 . A 12 GB Project with 1253 files took exactly 1 Hour. Initially it seemed faster for copying but it becomes slow. A 12 GB folder with 1253 files only takes around 8 minutes to copy via finder. Why is reaper taking this long ? I cannot understand. Surely this needs to be better for reaper users to collaborate with one another..

[svijayrathinam]

Developers pls.. Is there a solution for this?

[ÄxxlForce]

I'm also affected as well! The last backup took me over 48h so naturally I'm not backing up every day which makes me very nervous :-/

I'm pretty sure it has to do witch the small "copying..." dialog. At first everything seems fine but the more files are copied and the more dialogs are opened the WindowServer gets completely clogged to a point where the entire Mac is almost unusable. I also notices that 90% of the data is copied in a few minutes and it needs the other 47.5h to copy the remaining few bytes.

For comparison: coping the files via finder takes 7 minutes.

[svijayrathinam]

Thank God atleast I am not alone... Developers pls...take a look... Even today I had a project which only took 3 minutes to copy via finder.... But it took like 25 minutes to copy via save project as with media files.. Pls..help

[svijayrathinam]

To be honest rendering every item and converting to protools session via AATRANSLATOR seems faster than save project as with media items.. I would think collaborating with reaper users should be more easier than collaborating with protools users. But apparently not. Pls developers ... Need your help on this. Infact I went as far as asking X Ray M to develop a script to copy all media items. He also had similar issue.

[ÄxxlForce]

I've attached a screenshot of the 48h backup session I was talking about. As much as I love Reaper I have to admit that it's not productively usable in this state

Is there another way to make a backup of a project which ensures that all files that are being using in the project are included?

[Justin]

I've tested using 5.974rc2, on a High Sierra RMBP (mid-2014).

My test project has 1400 .wav files of varying sizes ranging from 2.5MB to 50MB, totaling 14GB. All of the media is stored in the same path as the project, and the project is on the startup disk (SSD).

Save-as with media, if saving to the project directory, is close to instant (since all of the media is there).

Save-as with media to a new path on the same disk takes about 30s.

Save-as with media to a new path on a network share is going a little slower than a copy with finder, but not much (20 minutes vs 17 minutes).

Is there something different here that's causing the slowdown for you? Maybe having the .wav files spread out in different paths?

[X-Raym]

Originally Posted by svijayrathinam

X Ray M to develop a script to copy all media items.

Yep, and I actually made it !


Here was the results on Windows, comparing Save As with a CMD script to copy the exact same file :






As you can see, way faster with CMD, even with just the copy part of the process (not the rebuild peak part).



I tried to port it to Mac script but it wasn't significatively good compared to the REAPER save as. Though, I have a 2011 Mac Book Air so it is kinda slow for every kind of tasks anyway.

[ÄxxlForce]

I never had problems with it on Windows. In my perception it's a problem in the macOS version. as I mentioned before I think it has to do with the modal dialogs displaying the copying status of each item. it seems that the more dialogs are opened the more it is slowed down.

yes my media files are located in different paths, that's why it is so important that I can make a backup which contains all the used files. btw I only use "copy all media into project ..." never "move all media into project ..."

maybe I should elaborate on my workflow: I have an existing project and I get sent additional tracks which I drag into the project. as soon as I do that the media paths a fragmented because the files are not actually copied into the project (which is a good idea because that way it's super fast). but at the end of the day I need to save the project with "copy all media into project..." in order to have a project that is self contained and doesn't have outside dependencies. it also gets rid of all unused media files.

that is such a crucial thing that I'm beginning to think I'm doing something wrong. How would you guys handle that? I know that there is the switch "Options: When importing, copy imported media to project media directory" but that won't save me if some media was accidentally imported without that flag turned on. I'd end up with a broken project.

edit: what would help me for example would be an action like "assert all used media files are in media directory and delete all unused media"

[Justin]

Save-as with copy shouldn't be that slow, not sure why you are seeing it as such. Especially if you're just incrementally saving some more media to the project. If you can come up with a list of steps that I can duplicate (maybe there's some specific combination of things), then maybe we can address it....

[svijayrathinam]

Hi Justin, here is what I am doing. Perhaps you can replicate the bug.

I am editing FX..so

1 drag and drop around 8000 files...via media explorer...just search for whatever files..as you wish..pls..make sure.. You have mixed file formats...some are wav, some are Flac some are Aiff...All these files are saved in NAS with SSD drives ( Synology NAS)

2 while drag and dropping..do not copy the files into project audio directory.

3 . Do a save project as with copy media files checked.. Into another drive...

This takes hours for me !!



If I select the same files and copy via finder... It only takes..less than 5 minutes...


Mac os Mojave cylinder Mac pro 6 core 64 GB RAM

[ÄxxlForce]

Originally Posted by Justin

Save-as with copy shouldn't be that slow, not sure why you are seeing it as such. Especially if you're just incrementally saving some more media to the project. If you can come up with a list of steps that I can duplicate (maybe there's some specific combination of things), then maybe we can address it....

ah no, if I'm doing a "proper" backup I save it to an entirely new path (another disk for example). this way I was always sure, that the backuped project contained all used media files and no unused. that used to be the last thing that I do after working on a project. I always found it very reliable and it didn't take too long.

I'll try to compose a list to reproduce it.

edit: btw only internal HDDs are used, no network drives whatsoever

[akademie]

Originally Posted by svijayrathinam

...
...
1 drag and drop around 8000 files...via media explorer...just search for whatever files..as you wish..pls..make sure.. You have mixed file formats...some are wav, some are Flac some are Aiff...All these files are saved in NAS with SSD drives ( Synology NAS)
3 . Do a save project as with copy media files checked.. Into another drive...
This takes hours for me !!

If I select the same files and copy via finder... It only takes..less than 5 minutes...
...

Hi svijayrathinam,
isn't it possible that you have also "Convert media.." option checked in "Save As.." dialogue window?? Then it would not only copy the media but also convert them from flac and aiff, etc.. That would describe the huge time difference against just straightforward copy process in Finder (without any conversions)

[svijayrathinam]

Originally Posted by akademie

Hi svijayrathinam,
isn't it possible that you have also "Convert media.." option checked in "Save As.." dialogue window?? Then it would not only copy the media but also convert them from flac and aiff, etc.. That would describe the huge time difference against just straightforward copy process in Finder (without any conversions)

No I don't have convert media turned on. I am only doing a simple copy.

[akademie]

Originally Posted by svijayrathinam

No I don't have convert media turned on. I am only doing a simple copy.

OK, it was just an idea that came to my mind when I read your steps about different file formats on sources.

[svijayrathinam]

Please developers...any updates on this? This is killing me...I had to copy just 2 GB project to another location. It took nearly 45 minutes. Pls this needs some attention from developers

[ÄxxlForce]

Ok, I tried to find anything out of the ordinary with my project but that's really all the steps it took me:

Step 1
Create a new Project on HDD1. Record a few GB of audio (mine has ~35GB). Maybe this is of interest: I use slip editing for the drums so I have several 100K of items in the project. I also utilize Subprojects to organize my project better. Never caused me problems before though (on Windows).

Step 2 (don't know if this is even necessary).
Import audio from HDD2. Default behavior doesn't copy the actual data just yet.

Step 3
Use "Save project as" to HDD3 with the option to copy all media as well.

Result
The copying gets slower and slower the more "copying..." dialogs are opened (and closed) and the window manager becomes virtually unresponsive. Can take days to complete if the mac didn't get stuck at all.

Expected Result
Copying via finder takes about 8 minutes this is what I would expect "Save as" to take as well.

[Justin]

Originally Posted by ÄxxlForce

Ok, I tried to find anything out of the ordinary with my project but that's really all the steps it took me:

Step 1
Create a new Project on HDD1. Record a few GB of audio (mine has ~35GB). Maybe this is of interest: I use slip editing for the drums so I have several 100K of items in the project. I also utilize Subprojects to organize my project better. Never caused me problems before though (on Windows).

Step 2 (don't know if this is even necessary).
Import audio from HDD2. Default behavior doesn't copy the actual data just yet.

Step 3
Use "Save project as" to HDD3 with the option to copy all media as well.

Result
The copying gets slower and slower the more "copying..." dialogs are opened (and closed) and the window manager becomes virtually unresponsive. Can take days to complete if the mac didn't get stuck at all.

Expected Result
Copying via finder takes about 8 minutes this is what I would expect "Save as" to take as well.

Can you email the .rpp file saved just before step 3 to support at cockos dot com (and include a link to this thread) ? Thanks!

[ÄxxlForce]

Thanks for looking into it! e-mail is sent I included all subproject .rpp

[Justin]

See latest dev build

[X-Raym]

v5.974+dev0419 - April 19 2019   + Project save: improve media copying speed [t=216526]

https://forum.cockos.com/showthread.php?t=219955

[ÄxxlForce]

This fixes the problem on my end! Dev(s), thank you so much for the swift solution, you're awesome!

[Gershy13]

I am having this issue, when i use the save as copy feature and copy media, it starts off fast for the first few tracks but then drops speed significantly after.

I am running Reaper 7 on windows 11. No matter the drive used it happens (external, network, internal ssd)