Go Back   Cockos Incorporated Forums > REAPER Forums > REAPER Bug Reports

Reply
 
Thread Tools Display Modes
Old 02-09-2021, 05:11 AM   #1
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default Scan all databases action rescans existing files (FIXED)

Hi,

Recently I made a startup script which scans all databases.

Now each time I open reaper it starts scanning files which have not changed in ages. I have databases of gamedata sounds which are years old, but reaper starts scanning through sounds and dialogue which have definitely not changed.

Even if I close and reopen reaper it will start scanning things.


Repro steps

Create a database from a folder with a large amount of samples in it. It needs to be a big library to show the issue, hundreds of thousands of files. Tested on a library with around 180k files.
When database has finished scanning, rescan the database

The scan will seemingly find new files each time. The number of files it "discovers" seems to increase with time. If you immediately scan again, it will be fairly quick, but the next day, it will find more files and take longer.

As I have a startup action which rescans all databases, I see activity daily from purchased libraries which have not changed.
Two of collegues have also noticed this issue. It's obvious when you have a lot of databases and you scan them all, it can take a while to get into reaper, and I often have to cancel the scan to get to work.

Last edited by Travesty; 02-16-2021 at 04:27 AM.
Travesty is offline   Reply With Quote
Old 02-16-2021, 04:33 AM   #2
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

Edited and added some repro steps. It's a longwinded repro unfortunately as it requires large databases to see the issue
Travesty is offline   Reply With Quote
Old 02-16-2021, 08:56 PM   #3
Justin
Administrator
 
Justin's Avatar
 
Join Date: Jan 2005
Location: NYC
Posts: 14,196
Default

It may be that just scanning the file system for new files takes a while? (if the file system paths are not in the OS cache, etc)
Justin is online now   Reply With Quote
Old 02-17-2021, 03:31 AM   #4
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

I don't think so, the databases which are being scanned are years old in some cases and have not changed. Plus we only noticed this in the past couple of months.

I just opened Reaper and it's scannning a foley archive which hasn't been added to in at least 2 years.
Travesty is offline   Reply With Quote
Old 02-18-2021, 02:54 PM   #5
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 13,101
Default

I think we've got this figured out, if you are able to test 6.24rc2 that would be appreciated!

https://forum.cockos.com/forumdisplay.php?f=37
schwa is offline   Reply With Quote
Old 02-19-2021, 08:58 AM   #6
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

Great, I'll check when it's finished scanning databases
Travesty is offline   Reply With Quote
Old 02-19-2021, 09:06 AM   #7
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

It's still happening unfortunately.

I'm noticing the same files flashing up each time though. I'll get a video and see if it is actually the same files.
Travesty is offline   Reply With Quote
Old 02-19-2021, 09:23 AM   #8
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

As far as I can tell it is the same files each time. I probably thought it was different files before, because I kept deleting old databases to speed up the scanning. I've got no more I can do without now.

I wonder if these databases have become corrupted in some way.
I can't see any similarities in the sounds themselves. Some are from last year, some from 5 years ago.
Some have bwf description metadata, some don't.
Travesty is offline   Reply With Quote
Old 02-19-2021, 09:56 AM   #9
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 13,101
Default

Could you send us one of the problem files, please? support at cockos.
schwa is offline   Reply With Quote
Old 02-19-2021, 10:06 AM   #10
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

I've sent one over, cheers
Travesty is offline   Reply With Quote
Old 02-19-2021, 10:14 AM   #11
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

It's not just wave files either, there are rpp files, rpp-bak, zip, reapeaks...
Travesty is offline   Reply With Quote
Old 02-19-2021, 10:24 AM   #12
schwa
Administrator
 
schwa's Avatar
 
Join Date: Mar 2007
Location: NY
Posts: 13,101
Default

Well, it doesn't seem to have anything to do with the particular file -- if I create a database with only that file, it scans once only. This is a super pain to debug because of the apparent dependency on the database containing tens of thousands of files.

Do you get the same behavior in 6.24rc2 whether you run "scan database for new files" or "scan all databases for new files" from the shortcut list context menu?
schwa is offline   Reply With Quote
Old 02-19-2021, 10:49 AM   #13
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

Yeah exactly, it's a pain

I'll give it a go. I was just opening reaper and letting the startup action run "scan all databases for new files"
Travesty is offline   Reply With Quote
Old 02-19-2021, 02:37 PM   #14
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

I just tried scan for new files on the oldest database, the one containing the file I sent, and I get the same issue.

Seven of the other databases are fine, and one shows a few files being scanned, but it is over quite quickly. This is the largest database, and it's local to this machine.

The one which is taking all the time is the oldest one I mentioned above. This is on a network drive accessed via vpn, though I don't think that would be an issue?

I could try deleting the database and recreating it, which might work, but then we wouldn't know what caused it in the first place.

There were other databases which had the issue, but I deleted those to try and speed up the scans.
Travesty is offline   Reply With Quote
Old 02-25-2021, 06:54 AM   #15
Travesty
Human being with feelings
 
Travesty's Avatar
 
Join Date: Nov 2014
Posts: 670
Default

I deleted the worst offending database and then recreated it. The issue still exists, so we can rule out corruption.

I found some of the files which keep popping up in the scan and made a db from just that folder. That reproduced the issue with much fewer files, but these were propriatory, so I can't share them. I narrowed this further to a freely available library of astronomical audio files.

https://www.sonidosdealma.org/en/

The website is broken at the moment, but I should be able to send the sounds over, they are open source. It's 3000 files but only 14MB size.

Creating a db from these reproduced the issue reliably, but only over the vpn on the network drive.

So I think either the issue is related to creating databases over the network somehow, or the latency is causing the issue to be visible, where it would be too quick to spot on a local drive.
Travesty is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 02:13 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.